2021-09-06T10:36:27.841 INFO:root:teuthology version: 1.1.0-df69d6db 2021-09-06T10:36:27.843 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi/6376787 --name soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi --owner scheduled_soumyakoduri@teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} -- /home/teuthworker/archive/soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi/6376787/orig.config.yaml 2021-09-06T10:36:27.885 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-09-06T10:36:27.940 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi/6376787 branch: wip-skoduri-dbstore-vstart description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} email: null first_in_suite: false job_id: '6376787' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi nuke-on-error: true os_type: centos os_version: '8.3' overrides: admin_socket: branch: wip-skoduri-dbstore-vstart ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 2227f2382f1df5117a62687acd2d99ea5eeb871e wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 2227f2382f1df5117a62687acd2d99ea5eeb871e rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-skoduri-dbstore-vstart sha1: 2227f2382f1df5117a62687acd2d99ea5eeb871e owner: scheduled_soumyakoduri@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 2227f2382f1df5117a62687acd2d99ea5eeb871e sleep_before_teardown: 0 suite: rgw suite_branch: wip-skoduri-dbstore-vstart suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_2227f2382f1df5117a62687acd2d99ea5eeb871e/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 2227f2382f1df5117a62687acd2d99ea5eeb871e targets: smithi094.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUTN53Z3n5QlqTDgk0NpEVg01dFbGUi6kCFNTkBAjTvi+FP0HPPz4m/nWJJWMgn9J9x87EXF9d5j4yhDZZUbFiKnCtE0dFqp7yDt+646ytY2pcOWqdnv3fMuK2YhdkRApWI6PATZkxRTQwnfDUxTWfHFjtREnNxDmZfc2W0YrsU3gCqByrMTHrycORc7y3OUyenla28nMmmYOIRAOJmwJi5dZBtxSK85jeYMra3DXk/yW3puzjQiyWMtUhdo4xEE/dnf9rAENBmVX7dCyo1Kei73l5gzG0b5dlipQcx3coHbU+P6YpHmRbxYRNFGX2LKiW2Qe9N0tS4EYmfK2kRsvcurg9UYzOof3BHPHYDAFygcudl9WTELEmtRTOYXltluuzCFxpnO6S8z+pfnExOIx9aPCm0dvzJlQyfOTFs+OggDmYQOPlQnoiZLEbfukHyPaig4GpWwscWbmWsOPTqwF2PPyOoATGx5+xfcHkmYmM26ahbjySb/iMipmGwvdWwZU= smithi139.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxOSAXu0ahF2fafU0XYIhsUwAZYZNMc0rU5nwzbFpNLJIohFw7h/mcnGZ+vhFkODxC9wceI+y+q0Ku77hekOeNq2jhxMbe9ursZc16VNyEGSg8FYaJQ/pxeHtRGF7kpKP6Tjq1qTHnRIUu+8TuWjnw7F85vyH64WvQORUNGmeH3OEz3lv+ZQGFLO5v2w1Vux/q42+sYA6KGNSFqXGfYDha3W3HUmS+d+K/z6ZIz1ZUGN4eCikpM2wL9gcirFUOKnCLxjgrDI9cTZY01z21PQG4HeYoaH7ZTFO2rWVkzAZLPhhHVRNzgLJe1fIuAcFG3GOplFJI4GEWg+VOQNsh4+SG1Mm1rrWZSYrwQ74bNDrFD8kCnLfNcKqpo0t2Qh1IqbPOAcrWoAqnoMFzOlh1kkc0WsF30uEhQVnHkIlF0yZfWhKuHx0gwG9+jotBkxUImCL1Pj1hzupFeFIMdhVY7PLurYFJtwqvKFXl7WeWoub2qHGq7S6tbv6oqxQlksp2M/E= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2021-09-06_10:28:39 tube: smithi user: soumyakoduri verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-09-06T10:36:27.941 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_2227f2382f1df5117a62687acd2d99ea5eeb871e/qa; will attempt to use it 2021-09-06T10:36:27.942 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_2227f2382f1df5117a62687acd2d99ea5eeb871e/qa/tasks 2021-09-06T10:36:27.942 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-09-06T10:36:27.944 INFO:teuthology.task.internal:Checking packages... 2021-09-06T10:36:27.969 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '2227f2382f1df5117a62687acd2d99ea5eeb871e' 2021-09-06T10:36:27.969 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-09-06T10:36:27.970 INFO:teuthology.packaging:ref: None 2021-09-06T10:36:27.970 INFO:teuthology.packaging:tag: None 2021-09-06T10:36:27.971 INFO:teuthology.packaging:branch: wip-skoduri-dbstore-vstart 2021-09-06T10:36:27.971 INFO:teuthology.packaging:sha1: 2227f2382f1df5117a62687acd2d99ea5eeb871e 2021-09-06T10:36:27.971 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-skoduri-dbstore-vstart 2021-09-06T10:36:28.132 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-7472.g2227f238 2021-09-06T10:36:28.133 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-09-06T10:36:28.144 INFO:teuthology.task.internal:no buildpackages task found 2021-09-06T10:36:28.144 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-09-06T10:36:28.259 INFO:teuthology.task.internal:Saving configuration 2021-09-06T10:36:28.274 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-09-06T10:36:28.293 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-06T10:36:28.311 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-06 10:29:38.014555', 'description': '/home/teuthworker/archive/soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi/6376787', 'locked_by': 'scheduled_soumyakoduri@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFi9A2J8qKraVNIy+1QdWHrF+2eGXIh0gBi4rycSmWWT59/zxLbN7+YqCZwv3V7de9NAzu1xhlYpopK5WosgK2U=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi094.front.sepia.ceph.com'} 2021-09-06T10:36:28.326 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-06 10:29:38.014577', 'description': '/home/teuthworker/archive/soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi/6376787', 'locked_by': 'scheduled_soumyakoduri@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPEr49d57dbzyDCoctH/g5kuYwWN6ftx6LQQ1NofKHAsUCw1DLIB7AjPnBZmIGmYfuukbs9YLYGw07puwdk7V2A=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi139.front.sepia.ceph.com'} 2021-09-06T10:36:28.327 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-09-06T10:36:28.407 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-09-06T10:36:28.407 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-09-06T10:36:28.407 INFO:teuthology.run_tasks:Running task console_log... 2021-09-06T10:36:28.528 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f121fcddd90>, signals=[15]) 2021-09-06T10:36:28.529 INFO:teuthology.run_tasks:Running task internal.connect... 2021-09-06T10:36:28.623 INFO:teuthology.task.internal:Opening connections... 2021-09-06T10:36:28.623 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2021-09-06T10:36:28.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T10:36:28.699 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2021-09-06T10:36:28.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T10:36:28.779 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-09-06T10:36:28.789 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2021-09-06T10:36:28.814 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2021-09-06T10:36:28.816 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2021-09-06T10:36:28.892 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Linux" 2021-09-06T10:36:28.892 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="8" 2021-09-06T10:36:28.893 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2021-09-06T10:36:28.893 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2021-09-06T10:36:28.894 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="8" 2021-09-06T10:36:28.894 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el8" 2021-09-06T10:36:28.894 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Linux 8" 2021-09-06T10:36:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2021-09-06T10:36:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-06T10:36:28.896 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2021-09-06T10:36:28.896 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-09-06T10:36:28.896 INFO:teuthology.orchestra.run.smithi094.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-09-06T10:36:28.897 INFO:teuthology.orchestra.run.smithi094.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-09-06T10:36:28.898 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2021-09-06T10:36:28.917 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2021-09-06T10:36:28.975 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2021-09-06T10:36:28.975 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2021-09-06T10:36:29.048 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Linux" 2021-09-06T10:36:29.049 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="8" 2021-09-06T10:36:29.049 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2021-09-06T10:36:29.049 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2021-09-06T10:36:29.049 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="8" 2021-09-06T10:36:29.050 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el8" 2021-09-06T10:36:29.050 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Linux 8" 2021-09-06T10:36:29.050 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2021-09-06T10:36:29.050 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-06T10:36:29.050 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2021-09-06T10:36:29.051 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-09-06T10:36:29.051 INFO:teuthology.orchestra.run.smithi139.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-09-06T10:36:29.051 INFO:teuthology.orchestra.run.smithi139.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-09-06T10:36:29.052 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2021-09-06T10:36:29.068 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-09-06T10:36:29.097 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-09-06T10:36:29.162 INFO:teuthology.task.internal:Checking for old test directory... 2021-09-06T10:36:29.163 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2021-09-06T10:36:29.168 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2021-09-06T10:36:29.198 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-09-06T10:36:29.236 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-09-06T10:36:29.236 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2021-09-06T10:36:29.241 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2021-09-06T10:36:29.279 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-09-06T10:36:29.339 INFO:teuthology.run_tasks:Running task kernel... 2021-09-06T10:36:29.388 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-09-06T10:36:29.389 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-09-06T10:36:29.390 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2021-09-06T10:36:29.408 INFO:teuthology.orchestra.run.smithi094.stdout:4.18.0-305.10.2.el8_4.x86_64 2021-09-06T10:36:29.409 INFO:teuthology.task.kernel:Running kernel on smithi094: 4.18.0-305.10.2.el8_4.x86_64 2021-09-06T10:36:29.410 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2021-09-06T10:36:31.769 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-8 - AppStream 30 MB/s | 8.8 MB 00:00 2021-09-06T10:36:33.279 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-8 - Base 45 MB/s | 5.6 MB 00:00 2021-09-06T10:36:34.125 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-8 - Extras 161 kB/s | 10 kB 00:00 2021-09-06T10:36:37.474 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Linux 8 - AppStream 2.7 MB/s | 8.8 MB 00:03 2021-09-06T10:45:10.267 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Linux 8 - BaseOS 11 kB/s | 5.6 MB 08:31 2021-09-06T10:45:11.625 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Linux 8 - Extras 18 kB/s | 10 kB 00:00 2021-09-06T10:45:11.789 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-8 - PowerTools 22 MB/s | 2.3 MB 00:00 2021-09-06T10:45:14.438 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2021-09-06T10:45:15.627 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 10 MB 00:01 2021-09-06T10:45:17.797 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 4.8 MB/s | 24 kB 00:00 2021-09-06T10:45:19.725 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-4.18.0-305.10.2.el8_4.x86_64 is already installed. 2021-09-06T10:45:19.726 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-09-06T10:45:20.031 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T10:45:20.032 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T10:45:20.032 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2021-09-06T10:45:20.033 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T10:45:20.033 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T10:45:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 5.9 M 2021-09-06T10:45:20.033 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2021-09-06T10:45:20.033 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 36 M 2021-09-06T10:45:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 28 M 2021-09-06T10:45:20.034 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T10:45:20.034 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T10:45:20.034 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T10:45:20.035 INFO:teuthology.orchestra.run.smithi094.stdout:Install 3 Packages 2021-09-06T10:45:20.035 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T10:45:20.036 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 70 M 2021-09-06T10:45:20.036 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 88 M 2021-09-06T10:45:20.036 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T10:45:20.424 INFO:teuthology.orchestra.run.smithi094.stdout:(1/3): kernel-4.18.0-305.12.1.el8_4.x86_64.rpm 15 MB/s | 5.9 MB 00:00 2021-09-06T10:45:21.135 INFO:teuthology.orchestra.run.smithi094.stdout:(2/3): kernel-core-4.18.0-305.12.1.el8_4.x86_64 33 MB/s | 36 MB 00:01 2021-09-06T10:45:21.464 INFO:teuthology.orchestra.run.smithi094.stdout:(3/3): kernel-modules-4.18.0-305.12.1.el8_4.x86 20 MB/s | 28 MB 00:01 2021-09-06T10:45:21.467 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T10:45:21.468 INFO:teuthology.orchestra.run.smithi094.stdout:Total 49 MB/s | 70 MB 00:01 2021-09-06T10:45:21.581 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T10:45:21.878 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T10:45:21.879 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T10:45:23.069 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T10:45:23.090 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T10:45:25.520 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T10:45:26.048 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-core-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-06T10:45:29.297 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-06T10:45:29.656 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-06T10:45:32.625 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-06T10:45:32.859 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-06T10:45:53.369 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-06T10:45:53.667 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-06T10:45:53.668 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-06T10:45:53.668 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-core-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-06T10:45:54.102 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-06T10:45:54.103 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T10:45:54.103 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T10:45:54.103 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:45:54.104 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core-4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:45:54.104 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:45:54.104 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T10:45:54.104 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T10:45:54.163 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel --last | head -n 1 2021-09-06T10:45:54.296 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-4.18.0-305.12.1.el8_4.x86_64 Mon 06 Sep 2021 10:45:32 AM UTC 2021-09-06T10:45:54.298 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:45:54.299 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.10.2.el8_4.x86_64 Expected: 4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:45:54.300 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2021-09-06T10:45:54.320 INFO:teuthology.orchestra.run.smithi139.stdout:4.18.0-305.10.2.el8_4.x86_64 2021-09-06T10:45:54.320 INFO:teuthology.task.kernel:Running kernel on smithi139: 4.18.0-305.10.2.el8_4.x86_64 2021-09-06T10:45:54.321 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum install -y kernel 2021-09-06T10:45:56.610 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - AppStream 37 MB/s | 8.8 MB 00:00 2021-09-06T10:45:58.138 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Base 38 MB/s | 5.6 MB 00:00 2021-09-06T10:45:58.977 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Extras 160 kB/s | 10 kB 00:00 2021-09-06T10:46:02.517 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Linux 8 - AppStream 2.5 MB/s | 8.8 MB 00:03 2021-09-06T10:56:55.304 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Linux 8 - BaseOS 8.7 kB/s | 5.6 MB 10:51 2021-09-06T10:56:56.604 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Linux 8 - Extras 20 kB/s | 10 kB 00:00 2021-09-06T10:56:56.774 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - PowerTools 20 MB/s | 2.3 MB 00:00 2021-09-06T10:56:59.600 INFO:teuthology.orchestra.run.smithi139.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2021-09-06T10:57:00.407 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 14 MB/s | 10 MB 00:00 2021-09-06T10:57:02.614 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2021-09-06T10:57:04.570 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-4.18.0-305.10.2.el8_4.x86_64 is already installed. 2021-09-06T10:57:04.570 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-09-06T10:57:04.877 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T10:57:04.878 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T10:57:04.878 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-09-06T10:57:04.878 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T10:57:04.879 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T10:57:04.879 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 5.9 M 2021-09-06T10:57:04.879 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-09-06T10:57:04.879 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 36 M 2021-09-06T10:57:04.880 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 28 M 2021-09-06T10:57:04.880 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T10:57:04.880 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T10:57:04.880 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T10:57:04.881 INFO:teuthology.orchestra.run.smithi139.stdout:Install 3 Packages 2021-09-06T10:57:04.881 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T10:57:04.882 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 70 M 2021-09-06T10:57:04.882 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 88 M 2021-09-06T10:57:04.882 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T10:57:04.993 INFO:teuthology.orchestra.run.smithi139.stdout:(1/3): kernel-4.18.0-305.12.1.el8_4.x86_64.rpm 52 MB/s | 5.9 MB 00:00 2021-09-06T10:57:05.648 INFO:teuthology.orchestra.run.smithi139.stdout:(2/3): kernel-modules-4.18.0-305.12.1.el8_4.x86 36 MB/s | 28 MB 00:00 2021-09-06T10:57:06.020 INFO:teuthology.orchestra.run.smithi139.stdout:(3/3): kernel-core-4.18.0-305.12.1.el8_4.x86_64 31 MB/s | 36 MB 00:01 2021-09-06T10:57:06.023 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T10:57:06.024 INFO:teuthology.orchestra.run.smithi139.stdout:Total 61 MB/s | 70 MB 00:01 2021-09-06T10:57:06.137 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T10:57:06.433 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T10:57:06.434 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T10:57:07.621 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T10:57:07.652 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T10:57:10.047 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T10:57:10.515 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-core-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-06T10:57:13.775 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-06T10:57:14.131 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-06T10:57:17.104 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-06T10:57:17.350 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-06T10:57:37.836 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-06T10:57:38.137 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-06T10:57:38.137 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-06T10:57:38.138 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-core-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-06T10:57:38.554 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-06T10:57:38.555 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T10:57:38.555 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T10:57:38.555 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:38.555 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core-4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:38.556 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:38.556 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T10:57:38.556 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T10:57:38.615 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel --last | head -n 1 2021-09-06T10:57:38.749 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-4.18.0-305.12.1.el8_4.x86_64 Mon 06 Sep 2021 10:57:17 AM UTC 2021-09-06T10:57:38.751 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:38.752 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.10.2.el8_4.x86_64 Expected: 4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:38.752 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-09-06T10:57:38.753 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-09-06T10:57:38.753 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-09-06T10:57:38.754 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2021-09-06T10:57:38.755 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2021-09-06T10:57:38.756 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel --last | head -n 1 2021-09-06T10:57:38.897 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-4.18.0-305.12.1.el8_4.x86_64 Mon 06 Sep 2021 10:45:32 AM UTC 2021-09-06T10:57:38.899 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:38.899 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2021-09-06T10:57:38.939 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2021-09-06T10:57:38.939 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2021-09-06T10:57:38.940 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.02 2021-09-06T10:57:38.940 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 90.el8 2021-09-06T10:57:38.940 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2021-09-06T10:57:38.940 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Wed 11 Aug 2021 06:38:04 PM UTC 2021-09-06T10:57:38.941 INFO:teuthology.orchestra.run.smithi094.stdout:Group : System Environment/Base 2021-09-06T10:57:38.941 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 9197207 2021-09-06T10:57:38.941 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2021-09-06T10:57:38.941 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-09-06T10:57:38.941 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-09-06T10:57:38.942 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-09-06T10:57:38.942 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : kbuilder.bsys.centos.org 2021-09-06T10:57:38.942 INFO:teuthology.orchestra.run.smithi094.stdout:Relocations : (not relocatable) 2021-09-06T10:57:38.942 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : CentOS BuildSystem 2021-09-06T10:57:38.943 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2021-09-06T10:57:38.943 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2021-09-06T10:57:38.943 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2021-09-06T10:57:38.943 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2021-09-06T10:57:38.944 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T10:57:38.944 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-09-06T10:57:38.944 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2021-09-06T10:57:38.944 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2021-09-06T10:57:38.944 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2021-09-06T10:57:38.945 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T10:57:38.945 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2021-09-06T10:57:38.946 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-09-06T10:57:38.946 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:38.947 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-09-06T10:57:39.736 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2021-09-06T10:57:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:done 2021-09-06T10:57:41.819 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2021-09-06T10:57:41.845 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.q2Q60zSqHV 2021-09-06T10:57:41.846 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.q2Q60zSqHV 2021-09-06T10:57:41.915 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.q2Q60zSqHV 2021-09-06T10:57:42.068 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.q2Q60zSqHV is 6KB 2021-09-06T10:57:42.117 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.q2Q60zSqHV 2021-09-06T10:57:42.131 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2021-09-06T10:57:42.225 INFO:teuthology.orchestra.run.smithi094.stdout:ecc66dffcd264e5181894e69c53ff13c-0-rescue.conf 2021-09-06T10:57:42.226 INFO:teuthology.orchestra.run.smithi094.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-240.el8.x86_64.conf 2021-09-06T10:57:42.226 INFO:teuthology.orchestra.run.smithi094.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.10.2.el8_4.x86_64.conf 2021-09-06T10:57:42.226 INFO:teuthology.orchestra.run.smithi094.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.12.1.el8_4.x86_64.conf 2021-09-06T10:57:42.227 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:42.332 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2021-09-06T10:57:42.334 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2021-09-06T10:57:42.335 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi139.front.sepia.ceph.com, path=None, version=distro) 2021-09-06T10:57:42.335 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel --last | head -n 1 2021-09-06T10:57:42.475 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-4.18.0-305.12.1.el8_4.x86_64 Mon 06 Sep 2021 10:57:17 AM UTC 2021-09-06T10:57:42.477 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:42.477 DEBUG:teuthology.orchestra.run.smithi139:> sudo rpm -qi grub2-tools 2021-09-06T10:57:42.508 INFO:teuthology.orchestra.run.smithi139.stdout:Name : grub2-tools 2021-09-06T10:57:42.508 INFO:teuthology.orchestra.run.smithi139.stdout:Epoch : 1 2021-09-06T10:57:42.508 INFO:teuthology.orchestra.run.smithi139.stdout:Version : 2.02 2021-09-06T10:57:42.509 INFO:teuthology.orchestra.run.smithi139.stdout:Release : 90.el8 2021-09-06T10:57:42.509 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: x86_64 2021-09-06T10:57:42.509 INFO:teuthology.orchestra.run.smithi139.stdout:Install Date: Wed 11 Aug 2021 06:38:04 PM UTC 2021-09-06T10:57:42.509 INFO:teuthology.orchestra.run.smithi139.stdout:Group : System Environment/Base 2021-09-06T10:57:42.510 INFO:teuthology.orchestra.run.smithi139.stdout:Size : 9197207 2021-09-06T10:57:42.510 INFO:teuthology.orchestra.run.smithi139.stdout:License : GPLv3+ 2021-09-06T10:57:42.510 INFO:teuthology.orchestra.run.smithi139.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-09-06T10:57:42.510 INFO:teuthology.orchestra.run.smithi139.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-09-06T10:57:42.511 INFO:teuthology.orchestra.run.smithi139.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-09-06T10:57:42.511 INFO:teuthology.orchestra.run.smithi139.stdout:Build Host : kbuilder.bsys.centos.org 2021-09-06T10:57:42.511 INFO:teuthology.orchestra.run.smithi139.stdout:Relocations : (not relocatable) 2021-09-06T10:57:42.511 INFO:teuthology.orchestra.run.smithi139.stdout:Packager : CentOS BuildSystem 2021-09-06T10:57:42.511 INFO:teuthology.orchestra.run.smithi139.stdout:Vendor : CentOS 2021-09-06T10:57:42.512 INFO:teuthology.orchestra.run.smithi139.stdout:URL : http://www.gnu.org/software/grub/ 2021-09-06T10:57:42.512 INFO:teuthology.orchestra.run.smithi139.stdout:Summary : Support tools for GRUB. 2021-09-06T10:57:42.512 INFO:teuthology.orchestra.run.smithi139.stdout:Description : 2021-09-06T10:57:42.512 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T10:57:42.513 INFO:teuthology.orchestra.run.smithi139.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-09-06T10:57:42.513 INFO:teuthology.orchestra.run.smithi139.stdout:customizable bootloader with modular architecture. It supports a rich 2021-09-06T10:57:42.513 INFO:teuthology.orchestra.run.smithi139.stdout:variety of kernel formats, file systems, computer architectures and 2021-09-06T10:57:42.513 INFO:teuthology.orchestra.run.smithi139.stdout:hardware devices. 2021-09-06T10:57:42.513 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T10:57:42.514 INFO:teuthology.orchestra.run.smithi139.stdout:This subpackage provides tools for support of all platforms. 2021-09-06T10:57:42.515 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-09-06T10:57:42.515 INFO:teuthology.task.kernel:Updating grub on smithi139 to boot 4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:42.516 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-09-06T10:57:43.862 INFO:teuthology.orchestra.run.smithi139.stderr:Generating grub configuration file ... 2021-09-06T10:57:46.401 INFO:teuthology.orchestra.run.smithi139.stderr:done 2021-09-06T10:57:46.403 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2021-09-06T10:57:46.434 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.maaVSbEarY 2021-09-06T10:57:46.435 DEBUG:teuthology.orchestra.run.smithi139:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.maaVSbEarY 2021-09-06T10:57:46.505 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.maaVSbEarY 2021-09-06T10:57:46.668 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.maaVSbEarY is 6KB 2021-09-06T10:57:46.719 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.maaVSbEarY 2021-09-06T10:57:46.736 DEBUG:teuthology.orchestra.run.smithi139:> sudo /bin/ls /boot/loader/entries || true 2021-09-06T10:57:46.805 INFO:teuthology.orchestra.run.smithi139.stdout:ecc66dffcd264e5181894e69c53ff13c-0-rescue.conf 2021-09-06T10:57:46.806 INFO:teuthology.orchestra.run.smithi139.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-240.el8.x86_64.conf 2021-09-06T10:57:46.806 INFO:teuthology.orchestra.run.smithi139.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.10.2.el8_4.x86_64.conf 2021-09-06T10:57:46.806 INFO:teuthology.orchestra.run.smithi139.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.12.1.el8_4.x86_64.conf 2021-09-06T10:57:46.808 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-set-default ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:57:46.962 DEBUG:teuthology.orchestra.run.smithi139:> sudo shutdown -r now 2021-09-06T10:58:16.995 INFO:teuthology.misc:Re-opening connections... 2021-09-06T10:58:16.995 INFO:teuthology.misc:trying to connect to ubuntu@smithi094.front.sepia.ceph.com 2021-09-06T10:58:16.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-06T10:58:16.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T10:58:35.339 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2021-09-06T10:58:35.341 INFO:teuthology.misc:trying to connect to ubuntu@smithi139.front.sepia.ceph.com 2021-09-06T10:58:35.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-06T10:58:35.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T10:59:35.408 DEBUG:teuthology.orchestra.remote:timed out 2021-09-06T10:59:35.409 DEBUG:teuthology.misc:waited 78.41393542289734 2021-09-06T10:59:36.410 INFO:teuthology.misc:trying to connect to ubuntu@smithi094.front.sepia.ceph.com 2021-09-06T10:59:36.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-06T10:59:36.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T10:59:36.701 DEBUG:teuthology.orchestra.run.smithi094:> true 2021-09-06T10:59:36.967 DEBUG:teuthology.misc:waited 79.9712336063385 2021-09-06T10:59:37.968 INFO:teuthology.misc:trying to connect to ubuntu@smithi139.front.sepia.ceph.com 2021-09-06T10:59:37.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-06T10:59:37.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T10:59:38.360 DEBUG:teuthology.orchestra.run.smithi139:> true 2021-09-06T10:59:38.635 DEBUG:teuthology.misc:waited 81.6394591331482 2021-09-06T10:59:39.636 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2021-09-06T10:59:39.636 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-305.12.1.el8_4.x86_64"... 2021-09-06T10:59:39.637 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2021-09-06T10:59:39.654 INFO:teuthology.orchestra.run.smithi094.stdout:4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:59:39.655 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-305.12.1.el8_4.x86_64 vs 4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:59:39.655 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-09-06T10:59:39.656 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-09-06T10:59:39.656 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2021-09-06T10:59:39.656 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-305.12.1.el8_4.x86_64"... 2021-09-06T10:59:39.657 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2021-09-06T10:59:39.674 INFO:teuthology.orchestra.run.smithi139.stdout:4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:59:39.675 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-305.12.1.el8_4.x86_64 vs 4.18.0-305.12.1.el8_4.x86_64 2021-09-06T10:59:39.675 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-09-06T10:59:39.677 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-09-06T10:59:40.678 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-09-06T10:59:40.679 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-06T10:59:40.768 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2021-09-06T10:59:40.784 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-09-06T10:59:40.785 DEBUG:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-06T10:59:40.866 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1 2021-09-06T10:59:40.878 INFO:teuthology.run_tasks:Running task internal.base... 2021-09-06T10:59:40.887 INFO:teuthology.task.internal:Creating test directory... 2021-09-06T10:59:40.887 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-06T10:59:40.890 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-06T10:59:40.907 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-09-06T10:59:40.919 INFO:teuthology.run_tasks:Running task internal.archive... 2021-09-06T10:59:40.953 INFO:teuthology.task.internal:Creating archive directory... 2021-09-06T10:59:40.954 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-06T10:59:40.956 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-06T10:59:40.987 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-09-06T10:59:40.995 INFO:teuthology.task.internal:Enabling coredump saving... 2021-09-06T10:59:40.995 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-06T10:59:41.021 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-06T10:59:41.046 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-06T10:59:41.050 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-06T10:59:41.057 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-06T10:59:41.060 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-06T10:59:41.062 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-09-06T10:59:41.071 INFO:teuthology.task.internal:Configuring sudo... 2021-09-06T10:59:41.071 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-06T10:59:41.101 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-06T10:59:41.127 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-09-06T10:59:41.138 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-09-06T10:59:41.138 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-06T10:59:41.167 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-06T10:59:41.182 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-06T10:59:41.254 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-06T10:59:41.317 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-06T10:59:41.396 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-06T10:59:41.437 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T10:59:41.438 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-06T10:59:41.502 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-06T10:59:41.540 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-06T10:59:41.600 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-06T10:59:41.677 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-06T10:59:41.726 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T10:59:41.726 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-06T10:59:41.791 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2021-09-06T10:59:41.794 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2021-09-06T10:59:41.839 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-09-06T10:59:41.890 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-09-06T10:59:42.287 INFO:teuthology.run_tasks:Running task internal.timer... 2021-09-06T10:59:42.297 INFO:teuthology.task.internal:Starting timer... 2021-09-06T10:59:42.298 INFO:teuthology.run_tasks:Running task pcp... 2021-09-06T10:59:42.344 INFO:teuthology.run_tasks:Running task selinux... 2021-09-06T10:59:42.380 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-09-06T10:59:42.381 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2021-09-06T10:59:42.430 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: [ OK ] 2021-09-06T10:59:42.432 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2021-09-06T10:59:42.494 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: [ OK ] 2021-09-06T10:59:42.495 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-09-06T10:59:42.495 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2021-09-06T10:59:42.519 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2021-09-06T10:59:42.519 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2021-09-06T10:59:42.555 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2021-09-06T10:59:42.556 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi139.front.sepia.ceph.com': 'permissive'} 2021-09-06T10:59:42.556 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-09-06T10:59:42.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T10:59:42.586 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-09-06T10:59:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T10:59:42.621 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-09-06T10:59:42.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2021-09-06T10:59:42.663 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2021-09-06T10:59:42.689 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-09-06T10:59:42.705 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-09-06T10:59:42.706 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-09-06T10:59:42.726 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-09-06T10:59:42.727 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com,smithi139.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-09-06T10:59:43.107 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-09-06T10:59:43.954 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-09-06T10:59:46.861 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-09-06T10:59:46.886 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:46 +0000 (0:00:00.042) 0:00:00.042 ****** 2021-09-06T10:59:46.899 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:46 +0000 (0:00:00.012) 0:00:00.054 ****** 2021-09-06T10:59:49.419 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi094.front.sepia.ceph.com] 2021-09-06T10:59:49.470 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:49 +0000 (0:00:02.571) 0:00:02.625 ****** 2021-09-06T10:59:49.744 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-09-06T10:59:49.746 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-06T10:59:49.767 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:49 +0000 (0:00:00.296) 0:00:02.922 ****** 2021-09-06T10:59:50.170 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-09-06T10:59:50.173 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-09-06T10:59:50.195 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:50 +0000 (0:00:00.427) 0:00:03.349 ****** 2021-09-06T10:59:50.264 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-09-06T10:59:50.265 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:50.286 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:50 +0000 (0:00:00.092) 0:00:03.441 ****** 2021-09-06T10:59:50.550 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi139.front.sepia.ceph.com] 2021-09-06T10:59:50.557 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T10:59:50.579 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:50 +0000 (0:00:00.293) 0:00:03.735 ****** 2021-09-06T10:59:50.620 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:50 +0000 (0:00:00.039) 0:00:03.775 ****** 2021-09-06T10:59:50.816 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-06T10:59:50.845 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:50 +0000 (0:00:00.225) 0:00:04.000 ****** 2021-09-06T10:59:51.154 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** ok: [smithi139.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-09-06T10:59:51.178 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:51 +0000 (0:00:00.332) 0:00:04.333 ****** 2021-09-06T10:59:51.251 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-09-06T10:59:51.253 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:51.273 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:51 +0000 (0:00:00.095) 0:00:04.428 ****** 2021-09-06T10:59:51.458 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-09-06T10:59:51.460 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T10:59:51.483 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:51 +0000 (0:00:00.210) 0:00:04.639 ****** 2021-09-06T10:59:51.511 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-09-06T10:59:51.513 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630925990.6226606-23957-162838283514378/source", "state": "file", "uid": 0} 2021-09-06T10:59:51.536 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:51 +0000 (0:00:00.052) 0:00:04.692 ****** 2021-09-06T10:59:51.965 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtC 2021-09-06T10:59:51.967 INFO:teuthology.task.ansible.out:toHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-09-06T10:59:52.126 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-09-06T10:59:52.127 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630925991.5352423-24114-208705421681775/source", "state": "file", "uid": 0} 2021-09-06T10:59:52.148 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:52 +0000 (0:00:00.611) 0:00:05.303 ****** 2021-09-06T10:59:52.481 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-09-06T10:59:52.708 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-09-06T10:59:52.710 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} PLAY [teuthology] ************************************************************** skipping: no hosts matched PLAY [testnodes] *************************************************************** TASK [Gathering Facts] ********************************************************* Monday 06 September 2021 10:59:52 +0000 (0:00:00.418) 0:00:05.722 ****** 2021-09-06T10:59:53.100 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] 2021-09-06T10:59:54.089 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] 2021-09-06T10:59:54.109 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** 2021-09-06T10:59:54.110 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:54 +0000 (0:00:01.542) 0:00:07.264 ****** 2021-09-06T10:59:54.134 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-06T10:59:54.148 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-06T10:59:54.151 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-06T10:59:54.330 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:54 +0000 (0:00:00.220) 0:00:07.484 ****** 2021-09-06T10:59:54.349 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:54 +0000 (0:00:00.020) 0:00:07.505 ****** 2021-09-06T10:59:54.937 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi094.front.sepia.ceph.com] 2021-09-06T10:59:54.990 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-09-06T10:59:54.992 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:54 +0000 (0:00:00.641) 0:00:08.146 ****** 2021-09-06T10:59:55.002 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] 2021-09-06T10:59:55.056 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:55 +0000 (0:00:00.065) 0:00:08.212 ****** 2021-09-06T10:59:55.724 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-09-06T10:59:55.726 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.291185", "end": "2021-09-06 10:59:55.701632", "rc": 0, "start": "2021-09-06 10:59:55.410447", "stderr": "", "stderr_lines": [], "stdout": "540ef3733cffec190e79eb6099e95cef354b655b", "stdout_lines": ["540ef3733cffec190e79eb6099e95cef354b655b"]} 2021-09-06T10:59:55.752 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:55 +0000 (0:00:00.695) 0:00:08.907 ****** 2021-09-06T10:59:55.760 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.313771", "end": "2021-09-06 10:59:55.724919", "rc": 0, "start": "2021-09-06 10:59:55.411148", "stderr": "", "stderr_lines": [], "stdout": "540ef3733cffec190e79eb6099e95cef354b655b", "stdout_lines": ["540ef3733cffec190e79eb6099e95cef354b655b"]} 2021-09-06T10:59:55.790 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:55 +0000 (0:00:00.038) 0:00:08.946 ****** 2021-09-06T10:59:55.953 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-09-06T10:59:55.955 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014460", "end": "2021-09-06 10:59:55.933922", "failed_when_result": false, "rc": 0, "start": "2021-09-06 10:59:55.919462", "stderr": "", "stderr_lines": [], "stdout": "c29274177b79f72740fc714fa28b55400cd0737f", "stdout_lines": ["c29274177b79f72740fc714fa28b55400cd0737f"]} 2021-09-06T10:59:55.976 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:55 +0000 (0:00:00.185) 0:00:09.132 ****** 2021-09-06T10:59:55.995 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.012643", "end": "2021-09-06 10:59:55.976802", "failed_when_result": false, "rc": 0, "start": "2021-09-06 10:59:55.964159", "stderr": "", "stderr_lines": [], "stdout": "c29274177b79f72740fc714fa28b55400cd0737f", "stdout_lines": ["c29274177b79f72740fc714fa28b55400cd0737f"]} 2021-09-06T10:59:56.024 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.047) 0:00:09.179 ****** 2021-09-06T10:59:56.041 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-09-06T10:59:56.043 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:56.065 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.040) 0:00:09.220 ****** 2021-09-06T10:59:56.093 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:56.116 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.051) 0:00:09.271 ****** 2021-09-06T10:59:56.134 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-09-06T10:59:56.135 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:56.159 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.042) 0:00:09.314 ****** 2021-09-06T10:59:56.187 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:56.212 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.052) 0:00:09.366 ****** 2021-09-06T10:59:56.229 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-09-06T10:59:56.231 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:56.256 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.045) 0:00:09.412 ****** 2021-09-06T10:59:56.275 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:56.305 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.048) 0:00:09.460 ****** 2021-09-06T10:59:56.325 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-09-06T10:59:56.327 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:56.352 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.046) 0:00:09.507 ****** 2021-09-06T10:59:56.391 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:56.419 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.066) 0:00:09.574 ****** 2021-09-06T10:59:56.425 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-09-06T10:59:56.428 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:56.457 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.037) 0:00:09.611 ****** 2021-09-06T10:59:56.488 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T10:59:56.513 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.057) 0:00:09.668 ****** 2021-09-06T10:59:56.690 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-09-06T10:59:56.692 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T10:59:56.713 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.200) 0:00:09.868 ****** 2021-09-06T10:59:56.752 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T10:59:56.775 INFO:teuthology.task.ansible.out:Monday 06 September 2021 10:59:56 +0000 (0:00:00.061) 0:00:09.930 ****** 2021-09-06T10:59:57.003 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-09-06T10:59:57.005 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-06T10:59:57.058 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-06T10:59:57.195 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-06T10:59:57.259 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-06T10:59:57.390 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-06T10:59:57.444 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-06T10:59:57.592 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-06T10:59:57.647 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-06T10:59:57.788 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-06T10:59:57.841 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-06T10:59:57.981 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-06T10:59:58.035 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-06T10:59:58.172 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-06T10:59:58.225 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-06T10:59:58.375 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-06T10:59:58.439 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-06T10:59:58.566 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-06T10:59:58.630 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-06T10:59:58.758 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-06T10:59:58.822 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-06T10:59:58.959 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-06T10:59:59.024 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-06T10:59:59.149 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-06T10:59:59.223 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-06T10:59:59.353 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-06T10:59:59.416 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-06T10:59:59.543 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-06T10:59:59.617 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-06T10:59:59.735 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-06T10:59:59.815 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-06T10:59:59.934 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-06T11:00:00.009 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-06T11:00:00.138 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-06T11:00:00.211 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-06T11:00:00.326 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-09-06T11:00:00.400 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-09-06T11:00:00.526 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-06T11:00:00.600 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-06T11:00:00.716 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-06T11:00:00.790 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-06T11:00:00.915 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-09-06T11:00:00.989 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-09-06T11:00:01.106 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-06T11:00:01.180 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-06T11:00:01.297 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-06T11:00:01.382 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-06T11:00:01.498 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-06T11:00:01.583 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-06T11:00:01.691 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-06T11:00:01.778 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-06T11:00:01.894 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-06T11:00:01.981 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-06T11:00:02.089 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-06T11:00:02.174 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-06T11:00:02.281 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-06T11:00:02.377 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-06T11:00:02.474 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-06T11:00:02.572 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-06T11:00:02.679 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-06T11:00:02.768 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-06T11:00:02.879 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-06T11:00:02.965 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-06T11:00:03.074 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-06T11:00:03.172 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-06T11:00:03.268 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-09-06T11:00:03.375 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-09-06T11:00:03.471 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-06T11:00:03.569 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-06T11:00:03.686 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-06T11:00:03.772 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-06T11:00:03.882 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-06T11:00:03.967 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-06T11:00:04.073 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-09-06T11:00:04.157 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-09-06T11:00:04.274 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-06T11:00:04.359 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-06T11:00:04.477 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-06T11:00:04.561 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-06T11:00:04.668 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-06T11:00:04.753 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-06T11:00:04.859 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-06T11:00:04.955 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-06T11:00:05.062 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-06T11:00:05.146 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-06T11:00:05.263 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-06T11:00:05.339 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-06T11:00:05.456 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-06T11:00:05.542 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-06T11:00:05.660 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-06T11:00:05.738 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-06T11:00:05.867 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-06T11:00:05.931 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-06T11:00:06.048 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-06T11:00:06.135 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-06T11:00:06.254 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-06T11:00:06.334 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-06T11:00:06.452 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-06T11:00:06.539 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-06T11:00:06.656 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-06T11:00:06.741 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-06T11:00:06.847 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-06T11:00:06.932 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-06T11:00:07.049 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-06T11:00:07.135 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-06T11:00:07.241 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-06T11:00:07.337 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-06T11:00:07.449 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-06T11:00:07.531 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-06T11:00:07.668 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-06T11:00:07.732 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-06T11:00:07.862 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-06T11:00:07.928 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-06T11:00:08.056 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-06T11:00:08.131 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-06T11:00:08.249 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-06T11:00:08.339 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-06T11:00:08.442 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-06T11:00:08.527 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-06T11:00:08.634 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-06T11:00:08.719 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-06T11:00:08.837 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-06T11:00:08.922 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-06T11:00:09.041 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-06T11:00:09.113 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-06T11:00:09.230 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-06T11:00:09.315 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-06T11:00:09.422 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-06T11:00:09.509 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-06T11:00:09.616 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-06T11:00:09.711 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-06T11:00:09.818 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-06T11:00:09.907 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-06T11:00:10.013 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-06T11:00:10.098 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-06T11:00:10.206 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-06T11:00:10.290 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-06T11:00:10.407 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-06T11:00:10.492 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-06T11:00:10.598 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-06T11:00:10.694 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-06T11:00:10.788 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-06T11:00:10.884 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-06T11:00:10.992 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-06T11:00:11.076 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-06T11:00:11.183 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-06T11:00:11.267 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-06T11:00:11.386 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-06T11:00:11.469 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-06T11:00:11.585 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-06T11:00:11.660 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-06T11:00:11.777 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-06T11:00:11.862 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-06T11:00:11.978 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-06T11:00:12.053 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-06T11:00:12.171 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-06T11:00:12.258 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-06T11:00:12.374 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-06T11:00:12.451 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-06T11:00:12.568 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-06T11:00:12.642 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-06T11:00:12.770 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-06T11:00:12.843 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-06T11:00:12.960 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-06T11:00:13.044 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-06T11:00:13.160 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-06T11:00:13.235 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-06T11:00:13.361 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-09-06T11:00:13.437 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-09-06T11:00:13.564 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-09-06T11:00:13.632 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-09-06T11:00:13.759 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-09-06T11:00:13.831 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-09-06T11:00:13.958 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-06T11:00:14.022 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-06T11:00:14.160 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-06T11:00:14.224 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-06T11:00:14.351 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-06T11:00:14.425 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-06T11:00:14.552 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-06T11:00:14.615 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-06T11:00:14.753 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-06T11:00:14.816 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-06T11:00:14.944 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-06T11:00:15.008 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-06T11:00:15.145 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-06T11:00:15.209 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-06T11:00:15.338 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-06T11:00:15.402 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-06T11:00:15.529 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-06T11:00:15.615 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-06T11:00:15.733 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-06T11:00:15.808 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-06T11:00:15.928 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-06T11:00:16.013 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-06T11:00:16.120 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-06T11:00:16.205 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-06T11:00:16.312 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-06T11:00:16.409 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-06T11:00:16.504 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-06T11:00:16.599 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-06T11:00:16.709 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-06T11:00:16.793 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-06T11:00:16.900 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-06T11:00:16.985 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-06T11:00:17.093 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-06T11:00:17.188 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-06T11:00:17.295 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-06T11:00:17.380 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-06T11:00:17.486 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-06T11:00:17.581 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-06T11:00:17.687 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-06T11:00:17.771 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-06T11:00:17.876 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-06T11:00:17.971 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-06T11:00:18.077 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-06T11:00:18.161 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-06T11:00:18.266 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-06T11:00:18.361 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-06T11:00:18.456 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-06T11:00:18.551 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-06T11:00:18.657 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-06T11:00:18.752 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-06T11:00:18.848 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-06T11:00:18.943 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-06T11:00:19.050 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-06T11:00:19.145 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-06T11:00:19.244 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-06T11:00:19.351 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-06T11:00:19.446 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-06T11:00:19.541 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-06T11:00:19.646 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-06T11:00:19.741 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-06T11:00:19.848 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-06T11:00:19.933 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-06T11:00:20.050 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-06T11:00:20.134 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-06T11:00:20.250 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-06T11:00:20.326 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-06T11:00:20.442 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-06T11:00:20.527 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-06T11:00:20.642 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-06T11:00:20.715 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-06T11:00:20.841 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-06T11:00:20.915 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-06T11:00:21.030 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-06T11:00:21.114 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-06T11:00:21.229 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-06T11:00:21.313 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-06T11:00:21.418 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-06T11:00:21.513 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-06T11:00:21.618 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-06T11:00:21.703 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-06T11:00:21.808 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-06T11:00:21.903 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-06T11:00:22.009 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-06T11:00:22.095 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-06T11:00:22.212 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-06T11:00:22.295 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-06T11:00:22.401 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-06T11:00:22.490 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-06T11:00:22.626 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-06T11:00:22.700 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-06T11:00:22.827 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-09-06T11:00:22.901 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-09-06T11:00:23.028 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-09-06T11:00:23.091 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-09-06T11:00:23.230 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-06T11:00:23.468 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-06T11:00:23.485 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-06T11:00:23.633 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-06T11:00:23.635 INFO:teuthology.task.ansible.out: 2021-09-06T11:00:23.687 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-06T11:00:23.837 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-06T11:00:23.891 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-06T11:00:24.039 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-06T11:00:24.082 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-06T11:00:24.239 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-06T11:00:24.282 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-06T11:00:24.429 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1238, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 1238} 2021-09-06T11:00:24.483 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1238, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 1238} 2021-09-06T11:00:24.631 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1239, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 1239} 2021-09-06T11:00:24.674 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1239, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 1239} 2021-09-06T11:00:24.822 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1240, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 1240} 2021-09-06T11:00:24.865 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1240, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 1240} 2021-09-06T11:00:25.012 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1241, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 1241} 2021-09-06T11:00:25.066 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1241, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 1241} 2021-09-06T11:00:25.214 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1242, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 1242} 2021-09-06T11:00:25.259 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1242, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 1242} 2021-09-06T11:00:25.408 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1243, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 1243} 2021-09-06T11:00:25.462 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1243, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 1243} 2021-09-06T11:00:25.609 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1244, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 1244} 2021-09-06T11:00:25.653 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1244, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 1244} 2021-09-06T11:00:25.802 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1245, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 1245} 2021-09-06T11:00:25.856 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1245, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 1245} 2021-09-06T11:00:26.005 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1246, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 1246} 2021-09-06T11:00:26.050 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1246, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 1246} 2021-09-06T11:00:26.198 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1247, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 1247} 2021-09-06T11:00:26.252 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1247, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 1247} 2021-09-06T11:00:26.401 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1248, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 1248} 2021-09-06T11:00:26.444 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1248, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 1248} 2021-09-06T11:00:26.594 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1249, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 1249} 2021-09-06T11:00:26.647 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1249, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 1249} 2021-09-06T11:00:26.796 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1250, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 1250} 2021-09-06T11:00:26.839 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1250, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 1250} 2021-09-06T11:00:26.987 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1251, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 1251} 2021-09-06T11:00:27.051 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1251, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 1251} 2021-09-06T11:00:27.203 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1252, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 1252} 2021-09-06T11:00:27.248 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1252, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 1252} 2021-09-06T11:00:27.386 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1253, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 1253} 2021-09-06T11:00:27.443 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1253, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 1253} 2021-09-06T11:00:28.993 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-09-06T11:00:29.111 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-09-06T11:00:30.792 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-09-06T11:00:30.898 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-09-06T11:00:32.423 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-09-06T11:00:32.595 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-09-06T11:00:32.728 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:00:32 +0000 (0:00:35.952) 0:00:45.883 ****** 2021-09-06T11:00:32.749 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:00:32 +0000 (0:00:00.020) 0:00:45.904 ****** 2021-09-06T11:00:32.792 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-09-06T11:00:32.818 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:00:32 +0000 (0:00:00.069) 0:00:45.973 ****** 2021-09-06T11:00:32.865 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:00:32 +0000 (0:00:00.044) 0:00:46.018 ****** 2021-09-06T11:00:33.111 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-09-06T11:00:33.112 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-06T11:00:33.115 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-06T11:00:33.116 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-09-06T11:00:33.118 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-09-06T11:00:33.119 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-09-06T11:00:33.121 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-09-06T11:00:33.123 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn" 2021-09-06T11:00:33.123 INFO:teuthology.task.ansible.out:: "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-09-06T11:00:33.125 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-09-06T11:00:33.127 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-09-06T11:00:33.129 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-09-06T11:00:33.132 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449 2021-09-06T11:00:33.135 INFO:teuthology.task.ansible.out:ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn 2021-09-06T11:00:33.137 INFO:teuthology.task.ansible.out:": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev 2021-09-06T11:00:33.140 INFO:teuthology.task.ansible.out: JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185 2021-09-06T11:00:33.141 INFO:teuthology.task.ansible.out:052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": " 2021-09-06T11:00:33.144 INFO:teuthology.task.ansible.out:pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}]}, "changed": false} 2021-09-06T11:00:33.146 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-06T11:00:33.148 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-06T11:00:33.150 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-09-06T11:00:33.151 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-09-06T11:00:33.153 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-09-06T11:00:33.155 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-09-06T11:00:33.157 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-09-06T11:00:33.158 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-09-06T11:00:33.160 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-09-06T11:00:33.162 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-09-06T11:00:33.163 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449 2021-09-06T11:00:33.166 INFO:teuthology.task.ansible.out:ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn 2021-09-06T11:00:33.168 INFO:teuthology.task.ansible.out:": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185 2021-09-06T11:00:33.170 INFO:teuthology.task.ansible.out:052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor- 2021-09-06T11:00:33.171 INFO:teuthology.task.ansible.out:mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": " 2021-09-06T11:00:33.173 INFO:teuthology.task.ansible.out:pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}]}, "changed": false} 2021-09-06T11:00:33.197 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:00:33 +0000 (0:00:00.333) 0:00:46.352 ****** 2021-09-06T11:00:33.224 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:00:33 +0000 (0:00:00.026) 0:00:46.378 ****** 2021-09-06T11:00:34.244 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-09-06T11:00:34.246 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"after": "540ef3733cffec190e79eb6099e95cef354b655b", "attempts": 1, "before": "540ef3733cffec190e79eb6099e95cef354b655b", "changed": false, "remote_url_changed": false} 2021-09-06T11:00:34.267 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:00:34 +0000 (0:00:01.044) 0:00:47.423 ****** 2021-09-06T11:00:34.287 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"after": "540ef3733cffec190e79eb6099e95cef354b655b", "attempts": 1, "before": "540ef3733cffec190e79eb6099e95cef354b655b", "changed": false, "remote_url_changed": false} 2021-09-06T11:00:34.311 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:00:34 +0000 (0:00:00.043) 0:00:47.466 ****** 2021-09-06T11:00:34.798 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-09-06T11:00:34.800 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item 2021-09-06T11:00:34.802 INFO:teuthology.task.ansible.out:", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-06T11:00:35.007 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-06T11:00:35.030 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-06T11:00:35.221 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-06T11:00:35.276 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-06T11:00:35.441 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-06T11:00:35.443 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-06T11:00:35.485 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-06T11:00:35.486 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-06T11:00:35.660 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-06T11:00:35.693 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-06T11:00:35.874 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-06T11:00:35.876 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-06T11:00:35.906 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-06T11:00:35.909 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-06T11:00:36.088 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-06T11:00:36.133 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-06T11:00:36.316 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-06T11:00:36.349 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-06T11:00:36.531 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-06T11:00:36.542 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:00:36.575 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1s 2021-09-06T11:00:36.577 INFO:teuthology.task.ansible.out:cr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:00:36.757 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-06T11:00:36.759 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-06T11:00:36.790 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-06T11:00:36.792 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-06T11:00:36.975 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-06T11:00:36.977 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:00:36.989 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:00:37.010 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-06T11:00:37.012 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-09-06T11:00:37.025 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:00:37.027 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:00:37.196 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-06T11:00:37.263 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-06T11:00:37.409 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-06T11:00:37.412 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-06T11:00:37.491 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-06T11:00:37.497 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-06T11:00:37.626 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-06T11:00:37.698 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-06T11:00:37.849 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-09-06T11:00:37.851 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-09-06T11:00:37.924 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-09-06T11:00:37.927 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-09-06T11:00:38.068 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-06T11:00:38.135 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-06T11:00:38.307 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-06T11:00:38.361 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-06T11:00:38.523 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-09-06T11:00:38.578 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-09-06T11:00:38.763 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-06T11:00:38.765 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-06T11:00:38.798 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-06T11:00:38.800 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-06T11:00:38.980 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-06T11:00:38.982 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-06T11:00:39.024 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-06T11:00:39.028 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-06T11:00:39.209 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-06T11:00:39.246 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-06T11:00:39.409 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-06T11:00:39.465 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-06T11:00:39.635 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2021-09-06T11:00:39.637 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-06T11:00:39.680 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2021-09-06T11:00:39.682 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-06T11:00:39.855 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-06T11:00:39.922 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-06T11:00:40.086 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-06T11:00:40.154 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-06T11:00:40.307 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-06T11:00:40.374 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-06T11:00:40.521 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-06T11:00:40.617 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-06T11:00:40.734 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-06T11:00:40.830 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-06T11:00:40.949 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-06T11:00:40.951 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-06T11:00:41.059 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-06T11:00:41.065 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-06T11:00:41.163 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-09-06T11:00:41.298 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-09-06T11:00:41.372 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-06T11:00:41.519 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-06T11:00:41.588 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-06T11:00:41.731 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-06T11:00:41.807 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-06T11:00:41.958 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-06T11:00:42.021 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-06T11:00:42.159 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-06T11:00:42.234 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-06T11:00:42.381 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-06T11:00:42.438 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-06T11:00:42.581 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-06T11:00:42.671 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-06T11:00:42.812 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-06T11:00:42.867 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-06T11:00:42.869 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-06T11:00:43.004 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-06T11:00:43.005 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-06T11:00:43.080 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-06T11:00:43.081 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-06T11:00:43.220 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-06T11:00:43.222 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-06T11:00:43.284 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-06T11:00:43.423 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-06T11:00:43.488 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-06T11:00:43.637 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-06T11:00:43.703 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-06T11:00:43.841 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-06T11:00:43.906 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-06T11:00:43.908 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-06T11:00:44.043 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-06T11:00:44.045 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-06T11:00:44.130 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-06T11:00:44.134 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-06T11:00:44.136 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-06T11:00:44.267 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-06T11:00:44.268 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-06T11:00:44.270 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-06T11:00:44.324 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-06T11:00:44.484 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-06T11:00:44.542 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-06T11:00:44.698 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-06T11:00:44.745 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-06T11:00:44.894 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-06T11:00:44.958 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-06T11:00:45.096 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-06T11:00:45.160 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-06T11:00:45.300 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-06T11:00:45.365 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-06T11:00:45.367 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-06T11:00:45.513 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-06T11:00:45.515 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-06T11:00:45.581 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-06T11:00:45.731 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-06T11:00:45.796 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-06T11:00:45.936 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-06T11:00:46.012 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-06T11:00:46.013 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-06T11:00:46.149 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-06T11:00:46.151 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-06T11:00:46.245 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-06T11:00:46.363 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-06T11:00:46.450 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-06T11:00:46.578 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-06T11:00:46.686 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-06T11:00:46.688 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-06T11:00:46.797 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-06T11:00:46.799 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-06T11:00:46.919 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-06T11:00:47.002 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-06T11:00:47.120 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-06T11:00:47.205 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-06T11:00:47.335 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-06T11:00:47.421 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-06T11:00:47.541 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-06T11:00:47.627 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-06T11:00:47.755 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-06T11:00:47.842 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-06T11:00:47.982 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-06T11:00:48.134 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-06T11:00:48.190 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-06T11:00:48.275 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-06T11:00:48.425 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-06T11:00:48.471 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-06T11:00:48.633 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-06T11:00:48.679 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-06T11:00:48.853 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-06T11:00:48.897 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-06T11:00:49.058 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-06T11:00:49.107 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-06T11:00:49.272 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-06T11:00:49.326 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-06T11:00:49.489 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-06T11:00:49.533 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-06T11:00:49.713 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-06T11:00:49.747 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-06T11:00:49.907 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-06T11:00:49.951 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-06T11:00:50.124 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-06T11:00:50.167 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-06T11:00:50.329 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-06T11:00:50.385 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-06T11:00:50.543 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-06T11:00:50.618 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-06T11:00:50.778 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-06T11:00:50.833 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-06T11:00:50.991 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-06T11:00:51.034 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-06T11:00:51.228 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-06T11:00:51.230 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-06T11:00:51.250 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-06T11:00:51.252 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-06T11:00:51.429 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-06T11:00:51.472 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-06T11:00:51.632 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-06T11:00:51.676 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-06T11:00:51.843 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-06T11:00:51.886 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-06T11:00:52.054 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-09-06T11:00:52.108 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-09-06T11:00:52.266 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-09-06T11:00:52.268 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-09-06T11:00:52.310 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-09-06T11:00:52.312 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-09-06T11:00:52.481 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-06T11:00:52.482 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-06T11:00:52.526 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-06T11:00:52.528 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-06T11:00:52.681 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-06T11:00:52.727 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-06T11:00:52.886 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-06T11:00:52.940 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-06T11:00:53.099 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-06T11:00:53.142 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-06T11:00:53.310 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-06T11:00:53.353 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-06T11:00:53.512 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-06T11:00:53.557 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-06T11:00:53.716 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-06T11:00:53.769 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-06T11:00:53.916 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-06T11:00:54.027 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-06T11:00:54.130 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-06T11:00:54.133 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-06T11:00:54.186 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-06T11:00:54.188 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-06T11:00:54.346 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-06T11:00:54.392 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-06T11:00:54.552 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-06T11:00:54.606 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-06T11:00:54.754 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-06T11:00:54.808 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-06T11:00:54.960 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-06T11:00:55.027 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-06T11:00:55.165 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-06T11:00:55.241 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-06T11:00:55.380 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-06T11:00:55.456 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-06T11:00:55.593 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-06T11:00:55.596 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-06T11:00:55.679 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-06T11:00:55.681 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-06T11:00:55.796 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-06T11:00:55.881 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-06T11:00:56.012 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-06T11:00:56.090 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-06T11:00:56.218 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-06T11:00:56.294 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-06T11:00:56.411 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-06T11:00:56.507 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-06T11:00:56.624 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-06T11:00:56.709 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-06T11:00:56.836 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-06T11:00:56.922 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-06T11:00:57.038 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-06T11:00:57.135 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-06T11:00:57.241 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-06T11:00:57.338 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-06T11:00:57.455 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-06T11:00:57.546 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-06T11:00:57.665 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-06T11:00:57.760 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-06T11:00:57.879 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-06T11:00:57.976 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-06T11:00:58.081 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-06T11:00:58.177 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-06T11:00:58.283 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-06T11:00:58.381 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-06T11:00:58.486 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-06T11:00:58.593 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-06T11:00:58.701 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-06T11:00:58.797 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-06T11:00:58.909 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-06T11:00:59.008 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-06T11:00:59.114 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-06T11:00:59.211 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-06T11:00:59.321 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-06T11:00:59.426 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-06T11:00:59.539 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-06T11:00:59.542 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-06T11:00:59.632 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-06T11:00:59.634 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-06T11:00:59.742 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-06T11:00:59.843 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-06T11:00:59.958 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-06T11:01:00.054 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-06T11:01:00.160 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-06T11:01:00.261 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-06T11:01:00.358 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-06T11:01:00.474 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-06T11:01:00.579 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-06T11:01:00.717 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-06T11:01:00.791 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-06T11:01:00.921 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-06T11:01:00.998 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-06T11:01:01.125 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-06T11:01:01.210 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-06T11:01:01.339 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-06T11:01:01.414 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-06T11:01:01.555 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-06T11:01:01.631 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-06T11:01:01.758 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-06T11:01:01.834 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-06T11:01:01.962 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-06T11:01:02.038 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-09-06T11:01:02.177 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-09-06T11:01:02.252 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-06T11:01:02.379 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-06T11:01:02.455 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-06T11:01:02.598 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-06T11:01:02.674 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-06T11:01:02.796 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-06T11:01:02.882 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-06T11:01:03.009 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-06T11:01:03.084 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-06T11:01:03.089 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-06T11:01:03.229 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-06T11:01:03.231 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-06T11:01:03.294 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-06T11:01:03.431 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-06T11:01:03.505 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-06T11:01:03.634 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-06T11:01:03.711 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-06T11:01:03.849 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-06T11:01:03.923 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-06T11:01:04.050 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-06T11:01:04.126 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-06T11:01:04.254 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-06T11:01:04.326 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-06T11:01:04.454 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-06T11:01:04.539 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-06T11:01:04.676 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-06T11:01:04.741 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-06T11:01:04.881 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-06T11:01:04.945 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-06T11:01:05.087 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-06T11:01:05.164 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-06T11:01:05.292 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-06T11:01:05.377 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-06T11:01:05.496 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-06T11:01:05.591 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-06T11:01:05.728 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-06T11:01:05.792 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-06T11:01:05.938 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-06T11:01:06.002 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-06T11:01:06.154 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-06T11:01:06.220 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-06T11:01:06.358 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-06T11:01:06.433 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-06T11:01:06.572 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-06T11:01:06.667 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-06T11:01:06.770 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-06T11:01:06.888 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-06T11:01:06.987 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-06T11:01:07.093 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-06T11:01:07.192 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-06T11:01:07.298 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-06T11:01:07.395 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-06T11:01:07.503 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-06T11:01:07.602 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:07 +0000 (0:00:33.290) 0:01:20.757 ****** 2021-09-06T11:01:07.612 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-06T11:01:07.718 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:07 +0000 (0:00:00.116) 0:01:20.873 ****** 2021-09-06T11:01:07.727 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-09-06T11:01:07.730 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.732 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.751 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.760 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.761 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.827 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.839 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.850 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.852 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.862 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.874 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.885 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.887 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:07.985 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-06T11:01:08.018 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.020 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.105 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-06T11:01:08.116 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.127 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.223 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-06T11:01:08.337 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-06T11:01:08.427 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-06T11:01:08.439 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.460 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.462 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.472 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.484 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.495 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.506 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.513 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.528 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.530 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.540 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.544 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.546 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-06T11:01:08.557 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.570 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef84 2021-09-06T11:01:08.574 INFO:teuthology.task.ansible.out:54742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.582 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.584 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.594 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.606 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.608 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.609 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.610 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.627 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.639 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.640 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.641 INFO:teuthology.task.ansible.out: 2021-09-06T11:01:08.643 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.643 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\ 2021-09-06T11:01:08.650 INFO:teuthology.task.ansible.out:n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.657 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.668 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.670 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.682 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.694 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.696 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.698 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.709 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.711 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.721 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.722 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.733 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.736 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.736 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.747 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.749 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.760 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.762 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.776 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.778 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.788 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.789 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.791 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.793 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.803 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.805 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.815 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.817 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.829 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.831 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.843 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.844 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.846 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.856 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.868 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.879 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.881 INFO:teuthology.task.ansible.out: skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.892 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.894 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.905 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.906 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.917 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.918 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.919 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.930 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.932 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.942 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.944 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.946 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.958 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.970 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.980 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.982 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.982 INFO:teuthology.task.ansible.out:skipping: [smithi139.fro 2021-09-06T11:01:08.984 INFO:teuthology.task.ansible.out:nt.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.995 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.997 INFO:teuthology.task.ansible.out: skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:08.997 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.008 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.010 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.021 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.024 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.033 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.035 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.048 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.049 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.059 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.061 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.073 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.075 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.084 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.086 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.097 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.098 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.099 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@ 2021-09-06T11:01:09.101 INFO:teuthology.task.ansible.out:fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.110 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.112 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.123 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.124 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.135 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.137 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.148 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.150 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.161 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.163 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.174 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.176 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.187 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.188 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.199 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.201 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.201 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.213 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.216 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.225 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.227 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.238 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.239 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.250 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.251 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.252 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.264 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.266 INFO:teuthology.task.ansible.out: skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.266 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.278 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.279 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.291 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.303 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.304 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.314 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.316 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.328 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.330 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.340 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.342 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.353 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.365 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.376 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.378 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.390 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.392 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.403 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.404 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.404 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.416 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.427 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.432 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.444 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.446 INFO:teuthology.task.ansible.out: skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.459 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.484 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.486 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.493 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'Aru 2021-09-06T11:01:09.495 INFO:teuthology.task.ansible.out:naSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.508 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop 2021-09-06T11:01:09.511 INFO:teuthology.task.ansible.out:_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.520 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.531 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.534 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.538 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.558 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.560 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.584 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:09 +0000 (0:00:01.865) 0:01:22.739 ****** 2021-09-06T11:01:09.586 INFO:teuthology.task.ansible.out: 2021-09-06T11:01:09.593 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.595 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.595 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.608 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.621 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.662 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:09 +0000 (0:00:00.077) 0:01:22.817 ****** 2021-09-06T11:01:09.669 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-09-06T11:01:09.671 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.696 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:09 +0000 (0:00:00.032) 0:01:22.850 ****** 2021-09-06T11:01:09.744 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:09.775 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:09 +0000 (0:00:00.076) 0:01:22.927 ****** 2021-09-06T11:01:09.978 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-09-06T11:01:09.980 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-06T11:01:10.065 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-06T11:01:10.174 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-06T11:01:10.259 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-06T11:01:10.365 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-06T11:01:10.462 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-06T11:01:10.560 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-06T11:01:10.655 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-06T11:01:10.773 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-06T11:01:10.849 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-06T11:01:10.953 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-06T11:01:11.039 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-06T11:01:11.152 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-06T11:01:11.233 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-06T11:01:11.344 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-06T11:01:11.432 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-06T11:01:11.534 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-06T11:01:11.619 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-06T11:01:11.727 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-06T11:01:11.814 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-06T11:01:11.937 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-06T11:01:12.011 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-06T11:01:12.130 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-06T11:01:12.206 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-06T11:01:12.322 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-06T11:01:12.410 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-06T11:01:12.517 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-06T11:01:12.603 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-06T11:01:12.710 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-06T11:01:12.789 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-06T11:01:12.908 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-06T11:01:12.992 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-06T11:01:13.099 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-06T11:01:13.184 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-06T11:01:13.291 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-06T11:01:13.375 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-06T11:01:13.484 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-06T11:01:13.572 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-06T11:01:13.674 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-06T11:01:13.766 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-06T11:01:13.873 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-06T11:01:13.961 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-06T11:01:14.061 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-06T11:01:14.159 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-06T11:01:14.244 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-06T11:01:14.353 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-06T11:01:14.453 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-06T11:01:14.550 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-06T11:01:14.647 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-06T11:01:14.742 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-06T11:01:14.839 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-06T11:01:14.938 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-06T11:01:15.031 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-06T11:01:15.128 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-06T11:01:15.219 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-06T11:01:15.313 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-06T11:01:15.420 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-06T11:01:15.508 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-06T11:01:15.617 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-06T11:01:15.702 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-06T11:01:15.810 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-06T11:01:15.906 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-06T11:01:16.005 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-06T11:01:16.090 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-06T11:01:16.187 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-06T11:01:16.293 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-06T11:01:16.388 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-06T11:01:16.484 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-06T11:01:16.568 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-06T11:01:16.675 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-06T11:01:16.759 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-06T11:01:16.865 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-06T11:01:16.950 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-06T11:01:17.056 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-06T11:01:17.140 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-06T11:01:17.246 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-06T11:01:17.331 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-06T11:01:17.447 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-06T11:01:17.522 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-06T11:01:17.643 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-06T11:01:17.719 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-06T11:01:17.825 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-06T11:01:17.910 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-06T11:01:18.016 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-06T11:01:18.104 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-06T11:01:18.211 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-06T11:01:18.297 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-06T11:01:18.405 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-06T11:01:18.493 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-06T11:01:18.601 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-06T11:01:18.686 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-06T11:01:18.793 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-06T11:01:18.891 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-06T11:01:18.993 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-06T11:01:19.089 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-06T11:01:19.184 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-06T11:01:19.269 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-06T11:01:19.327 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:19 +0000 (0:00:09.554) 0:01:32.482 ****** 2021-09-06T11:01:19.379 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-06T11:01:19.420 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:19 +0000 (0:00:00.092) 0:01:32.575 ****** 2021-09-06T11:01:19.884 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-09-06T11:01:19.885 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "checksum": "86e4a2dae5f62bb631669de9d1a6c6efe9b72935", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "5694a3125663295343f0bc40f8059adb", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630926079.4300387-9592-118527175584185/source", "state": "file", "uid": 0} 2021-09-06T11:01:19.912 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:19 +0000 (0:00:00.491) 0:01:33.066 ****** 2021-09-06T11:01:19.972 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "checksum": "86e4a2dae5f62bb631669de9d1a6c6efe9b72935", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "5694a3125663295343f0bc40f8059adb", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630926079.513019-9608-156760279742629/source", "state": "file", "uid": 0} 2021-09-06T11:01:19.995 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:19 +0000 (0:00:00.084) 0:01:33.151 ****** 2021-09-06T11:01:20.004 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-09-06T11:01:20.006 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => { "msg": "Host smithi094.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-09-06T11:01:20.033 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.036) 0:01:33.188 ****** 2021-09-06T11:01:20.062 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => { "msg": "Host smithi139.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-09-06T11:01:20.085 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.052) 0:01:33.240 ****** 2021-09-06T11:01:20.115 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-09-06T11:01:20.139 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.054) 0:01:33.294 ****** 2021-09-06T11:01:20.167 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-09-06T11:01:20.192 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.053) 0:01:33.347 ****** 2021-09-06T11:01:20.210 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-09-06T11:01:20.214 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:20.233 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.041) 0:01:33.388 ****** 2021-09-06T11:01:20.261 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:20.286 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.053) 0:01:33.442 ****** 2021-09-06T11:01:20.549 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-09-06T11:01:20.551 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.077915", "end": "2021-09-06 11:01:20.526057", "rc": 0, "start": "2021-09-06 11:01:20.448142", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-06T11:01:20.573 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.285) 0:01:33.728 ****** 2021-09-06T11:01:20.603 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.079152", "end": "2021-09-06 11:01:20.581143", "rc": 0, "start": "2021-09-06 11:01:20.501991", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-06T11:01:20.627 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.054) 0:01:33.782 ****** 2021-09-06T11:01:20.644 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:20.668 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.041) 0:01:33.823 ****** 2021-09-06T11:01:20.698 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:20.725 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.057) 0:01:33.881 ****** 2021-09-06T11:01:20.745 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:20.769 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.043) 0:01:33.924 ****** 2021-09-06T11:01:20.797 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:20.824 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.055) 0:01:33.979 ****** 2021-09-06T11:01:20.843 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-09-06T11:01:20.844 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:20.868 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.043) 0:01:34.023 ****** 2021-09-06T11:01:20.901 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:20.924 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.056) 0:01:34.079 ****** 2021-09-06T11:01:20.942 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-09-06T11:01:20.946 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:20.974 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:20 +0000 (0:00:00.049) 0:01:34.128 ****** 2021-09-06T11:01:21.002 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.029 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.054) 0:01:34.183 ****** 2021-09-06T11:01:21.046 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.070 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.041) 0:01:34.224 ****** 2021-09-06T11:01:21.098 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.122 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.053) 0:01:34.278 ****** 2021-09-06T11:01:21.141 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.142 INFO:teuthology.task.ansible.out: 2021-09-06T11:01:21.166 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.043) 0:01:34.321 ****** 2021-09-06T11:01:21.194 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.220 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.054) 0:01:34.375 ****** 2021-09-06T11:01:21.222 INFO:teuthology.task.ansible.out: 2021-09-06T11:01:21.251 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-09-06T11:01:21.252 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:21.276 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.056) 0:01:34.432 ****** 2021-09-06T11:01:21.295 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:21.297 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:21.322 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.045) 0:01:34.477 ****** 2021-09-06T11:01:21.361 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.388 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.066) 0:01:34.544 ****** 2021-09-06T11:01:21.407 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.432 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.043) 0:01:34.587 ****** 2021-09-06T11:01:21.473 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-09-06T11:01:21.475 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.506 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.065) 0:01:34.652 ****** 2021-09-06T11:01:21.515 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.544 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.043) 0:01:34.696 ****** 2021-09-06T11:01:21.580 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.603 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.061) 0:01:34.758 ****** 2021-09-06T11:01:21.622 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.646 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.042) 0:01:34.801 ****** 2021-09-06T11:01:21.686 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-09-06T11:01:21.688 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:21.711 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.064) 0:01:34.866 ****** 2021-09-06T11:01:21.740 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:21.765 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.053) 0:01:34.919 ****** 2021-09-06T11:01:21.813 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-09-06T11:01:21.815 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.840 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.075) 0:01:34.995 ****** 2021-09-06T11:01:21.848 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:21.882 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.042) 0:01:35.037 ****** 2021-09-06T11:01:21.923 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-09-06T11:01:21.925 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:21.951 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:21 +0000 (0:00:00.069) 0:01:35.106 ****** 2021-09-06T11:01:22.001 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:22.025 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.073) 0:01:35.180 ****** 2021-09-06T11:01:22.031 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-09-06T11:01:22.033 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.056 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.030) 0:01:35.210 ****** 2021-09-06T11:01:22.117 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.143 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.086) 0:01:35.297 ****** 2021-09-06T11:01:22.160 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-09-06T11:01:22.162 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.185 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.042) 0:01:35.340 ****** 2021-09-06T11:01:22.223 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.248 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.063) 0:01:35.403 ****** 2021-09-06T11:01:22.266 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-09-06T11:01:22.269 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.296 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.048) 0:01:35.452 ****** 2021-09-06T11:01:22.325 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.350 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.053) 0:01:35.505 ****** 2021-09-06T11:01:22.371 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.394 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.043) 0:01:35.549 ****** 2021-09-06T11:01:22.422 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.448 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.054) 0:01:35.603 ****** 2021-09-06T11:01:22.469 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:22.495 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.045) 0:01:35.649 ****** 2021-09-06T11:01:22.523 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:22.549 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.054) 0:01:35.703 ****** 2021-09-06T11:01:22.577 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-06T11:01:22.579 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.603 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.054) 0:01:35.758 ****** 2021-09-06T11:01:22.631 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.655 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.051) 0:01:35.810 ****** 2021-09-06T11:01:22.683 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-09-06T11:01:22.685 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.710 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.054) 0:01:35.865 ****** 2021-09-06T11:01:22.739 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.764 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.054) 0:01:35.920 ****** 2021-09-06T11:01:22.782 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-09-06T11:01:22.784 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.805 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.041) 0:01:35.961 ****** 2021-09-06T11:01:22.846 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.871 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.065) 0:01:36.026 ****** 2021-09-06T11:01:22.889 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-09-06T11:01:22.891 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.915 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.043) 0:01:36.070 ****** 2021-09-06T11:01:22.944 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:22.968 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:22 +0000 (0:00:00.053) 0:01:36.124 ****** 2021-09-06T11:01:22.988 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-09-06T11:01:22.990 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:23.014 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.045) 0:01:36.169 ****** 2021-09-06T11:01:23.044 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:23.068 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.054) 0:01:36.223 ****** 2021-09-06T11:01:23.100 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-09-06T11:01:23.102 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:23.133 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.062) 0:01:36.286 ****** 2021-09-06T11:01:23.160 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:23.191 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.058) 0:01:36.344 ****** 2021-09-06T11:01:23.208 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:23.231 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.042) 0:01:36.387 ****** 2021-09-06T11:01:23.260 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T11:01:23.284 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.052) 0:01:36.439 ****** 2021-09-06T11:01:23.303 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-09-06T11:01:23.304 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:23.329 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.043) 0:01:36.483 ****** 2021-09-06T11:01:23.361 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:23.388 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.058) 0:01:36.542 ****** 2021-09-06T11:01:23.407 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:23.431 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.044) 0:01:36.586 ****** 2021-09-06T11:01:23.432 INFO:teuthology.task.ansible.out: 2021-09-06T11:01:23.460 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:23.494 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.063) 0:01:36.650 ****** 2021-09-06T11:01:23.702 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-09-06T11:01:23.703 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T11:01:23.728 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.232) 0:01:36.882 ****** 2021-09-06T11:01:23.779 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T11:01:23.803 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:23 +0000 (0:00:00.076) 0:01:36.959 ****** 2021-09-06T11:01:24.531 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} ok: [smithi094.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-09-06T11:01:24.978 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-09-06T11:01:25.001 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:24 +0000 (0:00:01.197) 0:01:38.157 ****** 2021-09-06T11:01:25.011 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-09-06T11:01:25.038 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:25 +0000 (0:00:00.036) 0:01:38.193 ****** 2021-09-06T11:01:25.547 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-09-06T11:01:25.549 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-09-06T11:01:25.580 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-09-06T11:01:25.988 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-09-06T11:01:26.015 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.975) 0:01:39.169 ****** 2021-09-06T11:01:26.034 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-09-06T11:01:26.060 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.046) 0:01:39.215 ****** 2021-09-06T11:01:26.088 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.112 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.051) 0:01:39.267 ****** 2021-09-06T11:01:26.142 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.166 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.054) 0:01:39.321 ****** 2021-09-06T11:01:26.183 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-06T11:01:26.186 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.208 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.041) 0:01:39.363 ****** 2021-09-06T11:01:26.239 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.262 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.054) 0:01:39.417 ****** 2021-09-06T11:01:26.281 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.305 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.043) 0:01:39.461 ****** 2021-09-06T11:01:26.333 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.358 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.051) 0:01:39.512 ****** 2021-09-06T11:01:26.376 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-09-06T11:01:26.377 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.400 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.042) 0:01:39.555 ****** 2021-09-06T11:01:26.427 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.451 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.051) 0:01:39.606 ****** 2021-09-06T11:01:26.480 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-09-06T11:01:26.481 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.482 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.493 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.519 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.067) 0:01:39.674 ****** 2021-09-06T11:01:26.537 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.539 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.550 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.553 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.578 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.059) 0:01:39.734 ****** 2021-09-06T11:01:26.608 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.633 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.054) 0:01:39.788 ****** 2021-09-06T11:01:26.651 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.679 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.046) 0:01:39.834 ****** 2021-09-06T11:01:26.706 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-09-06T11:01:26.708 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.733 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.054) 0:01:39.888 ****** 2021-09-06T11:01:26.751 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.776 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.042) 0:01:39.931 ****** 2021-09-06T11:01:26.805 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-09-06T11:01:26.807 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.831 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.055) 0:01:39.986 ****** 2021-09-06T11:01:26.833 INFO:teuthology.task.ansible.out: 2021-09-06T11:01:26.860 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.884 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.053) 0:01:40.040 ****** 2021-09-06T11:01:26.903 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-09-06T11:01:26.905 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.933 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.048) 0:01:40.088 ****** 2021-09-06T11:01:26.963 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:26.990 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:26 +0000 (0:00:00.056) 0:01:40.145 ****** 2021-09-06T11:01:27.008 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-09-06T11:01:27.014 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:27.032 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:27 +0000 (0:00:00.042) 0:01:40.187 ****** 2021-09-06T11:01:27.062 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:27.087 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:27 +0000 (0:00:00.055) 0:01:40.243 ****** 2021-09-06T11:01:27.516 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-09-06T11:01:27.518 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.233228", "end": "2021-09-06 11:01:27.500335", "rc": 0, "start": "2021-09-06 11:01:27.267107", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.295288", "end": "2021-09-06 11:01:27.500369", "rc": 0, "start": "2021-09-06 11:01:27.205081", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:01:27.542 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:27 +0000 (0:00:00.454) 0:01:40.697 ****** 2021-09-06T11:01:27.566 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:27 +0000 (0:00:00.020) 0:01:40.718 ****** 2021-09-06T11:01:31.546 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-09-06T11:01:31.548 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:01:31.577 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:31 +0000 (0:00:04.014) 0:01:44.732 ****** 2021-09-06T11:01:31.596 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:01:31.623 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:31 +0000 (0:00:00.045) 0:01:44.778 ****** 2021-09-06T11:01:31.654 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-09-06T11:01:31.656 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:31.678 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:31 +0000 (0:00:00.055) 0:01:44.833 ****** 2021-09-06T11:01:31.695 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:31.722 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:31 +0000 (0:00:00.043) 0:01:44.877 ****** 2021-09-06T11:01:31.750 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-09-06T11:01:31.752 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:31.779 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:31 +0000 (0:00:00.053) 0:01:44.931 ****** 2021-09-06T11:01:31.794 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:31.821 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:31 +0000 (0:00:00.043) 0:01:44.974 ****** 2021-09-06T11:01:31.849 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:31.874 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:31 +0000 (0:00:00.055) 0:01:45.029 ****** 2021-09-06T11:01:31.907 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:31.934 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:31 +0000 (0:00:00.059) 0:01:45.088 ****** 2021-09-06T11:01:31.953 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:31.978 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:31 +0000 (0:00:00.042) 0:01:45.131 ****** 2021-09-06T11:01:32.008 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:32.034 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:32 +0000 (0:00:00.056) 0:01:45.188 ****** 2021-09-06T11:01:32.488 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-09-06T11:01:32.490 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-09-06T11:01:32.514 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:32 +0000 (0:00:00.480) 0:01:45.669 ****** 2021-09-06T11:01:32.553 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-09-06T11:01:32.579 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:32 +0000 (0:00:00.062) 0:01:45.731 ****** 2021-09-06T11:01:34.105 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-09-06T11:01:34.107 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:01:34.129 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:34 +0000 (0:00:01.552) 0:01:47.284 ****** 2021-09-06T11:01:34.179 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:01:34.207 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:34 +0000 (0:00:00.075) 0:01:47.360 ****** 2021-09-06T11:01:34.679 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-06T11:01:34.682 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-09-06T11:01:34.729 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-09-06T11:01:35.111 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-09-06T11:01:35.179 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-09-06T11:01:35.526 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-09-06T11:01:35.553 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:35 +0000 (0:00:01.348) 0:01:48.708 ****** 2021-09-06T11:01:35.604 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-09-06T11:01:35.633 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:35 +0000 (0:00:00.079) 0:01:48.788 ****** 2021-09-06T11:01:35.748 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-09-06T11:01:35.750 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T11:01:35.775 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:35 +0000 (0:00:00.141) 0:01:48.930 ****** 2021-09-06T11:01:35.825 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T11:01:35.859 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:35 +0000 (0:00:00.084) 0:01:49.014 ****** 2021-09-06T11:01:36.268 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-06T11:01:36.269 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-09-06T11:01:36.357 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-09-06T11:01:36.711 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-09-06T11:01:36.778 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-09-06T11:01:37.115 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-09-06T11:01:37.139 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:37 +0000 (0:00:01.280) 0:01:50.295 ****** 2021-09-06T11:01:37.188 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-09-06T11:01:37.215 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:37 +0000 (0:00:00.075) 0:01:50.370 ****** 2021-09-06T11:01:37.222 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-09-06T11:01:37.227 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:37.251 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:37 +0000 (0:00:00.034) 0:01:50.405 ****** 2021-09-06T11:01:37.290 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:37.315 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:37 +0000 (0:00:00.064) 0:01:50.470 ****** 2021-09-06T11:01:37.336 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-09-06T11:01:37.338 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-09-06T11:01:37.363 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:37 +0000 (0:00:00.045) 0:01:50.516 ****** 2021-09-06T11:01:37.400 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-09-06T11:01:37.429 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:37 +0000 (0:00:00.062) 0:01:50.578 ****** 2021-09-06T11:01:37.609 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-09-06T11:01:37.611 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-06T11:01:37.634 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:37 +0000 (0:00:00.211) 0:01:50.789 ****** 2021-09-06T11:01:37.672 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-06T11:01:37.696 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:37 +0000 (0:00:00.061) 0:01:50.851 ****** 2021-09-06T11:01:37.736 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:37.757 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:37.759 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:37.779 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:37.804 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:37 +0000 (0:00:00.107) 0:01:50.959 ****** 2021-09-06T11:01:37.812 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:37.823 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:37.834 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:37.860 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:37 +0000 (0:00:00.056) 0:01:51.015 ****** 2021-09-06T11:01:39.453 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-09-06T11:01:39.455 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:01:39.477 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:39 +0000 (0:00:01.615) 0:01:52.631 ****** 2021-09-06T11:01:39.484 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:01:39.510 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:39 +0000 (0:00:00.033) 0:01:52.665 ****** 2021-09-06T11:01:39.548 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-09-06T11:01:39.550 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:39.573 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:39 +0000 (0:00:00.063) 0:01:52.728 ****** 2021-09-06T11:01:39.593 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:39.619 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:39 +0000 (0:00:00.045) 0:01:52.774 ****** 2021-09-06T11:01:40.066 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-09-06T11:01:40.069 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-09-06T11:01:40.091 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:40 +0000 (0:00:00.472) 0:01:53.247 ****** 2021-09-06T11:01:40.131 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-09-06T11:01:40.157 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:40 +0000 (0:00:00.065) 0:01:53.312 ****** 2021-09-06T11:01:40.605 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-09-06T11:01:40.630 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:40 +0000 (0:00:00.472) 0:01:53.785 ****** 2021-09-06T11:01:40.676 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-09-06T11:01:40.694 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:40 +0000 (0:00:00.064) 0:01:53.849 ****** 2021-09-06T11:01:40.724 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-09-06T11:01:40.726 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:40.735 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:40.759 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:40 +0000 (0:00:00.064) 0:01:53.913 ****** 2021-09-06T11:01:40.776 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:40.787 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-09-06T11:01:40.821 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:40 +0000 (0:00:00.057) 0:01:53.971 ****** 2021-09-06T11:01:40.994 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T11:01:41.017 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:41 +0000 (0:00:00.201) 0:01:54.172 ****** 2021-09-06T11:01:41.056 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T11:01:41.081 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:41 +0000 (0:00:00.063) 0:01:54.236 ****** 2021-09-06T11:01:41.304 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi094.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-09-06T11:01:41.324 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:41 +0000 (0:00:00.242) 0:01:54.479 ****** 2021-09-06T11:01:41.353 INFO:teuthology.task.ansible.out:fatal: [smithi139.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-09-06T11:01:41.376 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:41 +0000 (0:00:00.052) 0:01:54.531 ****** 2021-09-06T11:01:41.393 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-09-06T11:01:41.395 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:41.418 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:41 +0000 (0:00:00.041) 0:01:54.573 ****** 2021-09-06T11:01:41.456 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:01:41.479 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:41 +0000 (0:00:00.061) 0:01:54.634 ****** 2021-09-06T11:01:41.917 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-09-06T11:01:41.918 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-09-06T11:01:41.943 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:41 +0000 (0:00:00.461) 0:01:55.096 ****** 2021-09-06T11:01:41.981 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-09-06T11:01:42.004 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:41 +0000 (0:00:00.063) 0:01:55.159 ****** 2021-09-06T11:01:42.620 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1251", "ExecMainStartTimestamp": "Mon 2021-09-06 10:59:21 UTC", "ExecMainStartTimestampMonotonic": "20428709", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Mon 2021-09-06 10:59:21 UTC] ; stop_time=[n/a] ; pid=1251 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "984", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "1844674407370955 2021-09-06T11:01:42.622 INFO:teuthology.task.ansible.out:1615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "1251", "MemoryAccounting": "yes", "MemoryCurrent": "684032", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "on-abort", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no 2021-09-06T11:01:42.624 INFO:teuthology.task.ansible.out:", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Type": "simple", "UID": "988", "UMask": "0022", "WatchdogTimestamp": "Mon 2021-09-06 10:59:21 UTC", "WatchdogTimestampMonotonic": "20429088", "WatchdogUSec": "0"}} ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1244", "ExecMainStartTimestamp": "Mon 2021-09-06 10:59:26 UTC", "ExecMainStartTimestampMonotonic": "20498674", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Mon 2021-09-06 10:59:26 UTC] ; stop_time=[n/a] ; pid=1244 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "984", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", 2021-09-06T11:01:42.626 INFO:teuthology.task.ansible.out: "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "1244", "MemoryAccounting": "yes", "MemoryCurrent": "679936", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "on-abort", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", " 2021-09-06T11:01:42.627 INFO:teuthology.task.ansible.out:StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Type": "simple", "UID": "988", "UMask": "0022", "WatchdogTimestamp": "Mon 2021-09-06 10:59:26 UTC", "WatchdogTimestampMonotonic": "20498784", "WatchdogUSec": "0"}} 2021-09-06T11:01:42.645 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:42 +0000 (0:00:00.641) 0:01:55.801 ****** 2021-09-06T11:01:42.673 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:42 +0000 (0:00:00.027) 0:01:55.828 ****** 2021-09-06T11:01:42.876 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001610", "end": "2021-09-06 11:01:42.848545", "rc": 0, "start": "2021-09-06 11:01:42.846935", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-09-06T11:01:42.899 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:42 +0000 (0:00:00.225) 0:01:56.054 ****** 2021-09-06T11:01:42.907 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001787", "end": "2021-09-06 11:01:42.883346", "rc": 0, "start": "2021-09-06 11:01:42.881559", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-09-06T11:01:42.932 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:42 +0000 (0:00:00.033) 0:01:56.087 ****** 2021-09-06T11:01:44.487 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-09-06T11:01:44.488 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:01:44.512 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:44 +0000 (0:00:01.578) 0:01:57.666 ****** 2021-09-06T11:01:44.518 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:01:44.545 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:01:44 +0000 (0:00:00.034) 0:01:57.700 ****** 2021-09-06T11:02:21.746 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-09-06T11:02:21.771 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:21 +0000 (0:00:37.225) 0:02:34.926 ****** 2021-09-06T11:02:22.123 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-09-06T11:02:22.147 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:22 +0000 (0:00:00.376) 0:02:35.302 ****** 2021-09-06T11:02:22.527 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-09-06T11:02:22.529 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-09-06T11:02:22.550 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:22 +0000 (0:00:00.402) 0:02:35.705 ****** 2021-09-06T11:02:22.672 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-09-06T11:02:22.697 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:22 +0000 (0:00:00.147) 0:02:35.852 ****** 2021-09-06T11:02:37.483 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-09-06T11:02:37.484 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:14.683779", "end": "2021-09-06 11:02:37.454376", "failed_when_result": false, "rc": 0, "start": "2021-09-06 11:02:22.770597", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-09-06T11:02:37.508 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:37 +0000 (0:00:14.810) 0:02:50.663 ****** 2021-09-06T11:02:37.790 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:14.850086", "end": "2021-09-06 11:02:37.767529", "failed_when_result": false, "rc": 0, "start": "2021-09-06 11:02:22.917443", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-09-06T11:02:37.814 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:37 +0000 (0:00:00.305) 0:02:50.969 ****** 2021-09-06T11:02:38.065 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "md5sum": "286c682fc73965c17876891a4581da3e", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630926157.6204064-26468-236404574163008/source", "state": "file", "uid": 0} 2021-09-06T11:02:38.090 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:38 +0000 (0:00:00.276) 0:02:51.246 ****** 2021-09-06T11:02:38.341 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-09-06T11:02:38.343 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.029514", "end": "2021-09-06 11:02:38.322348", "rc": 0, "start": "2021-09-06 11:02:38.292834", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:02:38.363 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:38 +0000 (0:00:00.273) 0:02:51.519 ****** 2021-09-06T11:02:38.371 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-09-06T11:02:38.373 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "md5sum": "286c682fc73965c17876891a4581da3e", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630926157.9074345-26517-151604682155825/source", "state": "file", "uid": 0} 2021-09-06T11:02:38.396 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:38 +0000 (0:00:00.032) 0:02:51.551 ****** 2021-09-06T11:02:38.624 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-09-06T11:02:38.625 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.019844", "end": "2021-09-06 11:02:38.600024", "rc": 0, "start": "2021-09-06 11:02:38.580180", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:02:38.656 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:38 +0000 (0:00:00.259) 0:02:51.811 ****** 2021-09-06T11:02:38.663 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-09-06T11:02:38.664 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.006711", "end": "2021-09-06 11:02:38.615309", "rc": 0, "start": "2021-09-06 11:02:38.608598", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:02:38.690 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:38 +0000 (0:00:00.034) 0:02:51.846 ****** 2021-09-06T11:02:38.939 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-09-06T11:02:38.940 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.011574", "end": "2021-09-06 11:02:38.920611", "rc": 0, "start": "2021-09-06 11:02:38.909037", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:02:38.961 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:38 +0000 (0:00:00.270) 0:02:52.116 ****** 2021-09-06T11:02:53.662 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-09-06T11:02:53.664 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:14.768308", "end": "2021-09-06 11:02:53.641740", "rc": 0, "start": "2021-09-06 11:02:38.873432", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:02:53.686 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:53 +0000 (0:00:14.724) 0:03:06.841 ****** 2021-09-06T11:02:53.926 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-06T11:02:53.928 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-09-06T11:02:53.949 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:53 +0000 (0:00:00.263) 0:03:07.104 ****** 2021-09-06T11:02:53.988 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-09-06T11:02:53.990 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:14.794023", "end": "2021-09-06 11:02:53.961339", "rc": 0, "start": "2021-09-06 11:02:39.167316", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:02:54.024 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.074) 0:03:07.179 ****** 2021-09-06T11:02:54.042 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-09-06T11:02:54.044 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-09-06T11:02:54.073 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.049) 0:03:07.229 ****** 2021-09-06T11:02:54.164 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-09-06T11:02:54.165 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-06T11:02:54.187 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.113) 0:03:07.342 ****** 2021-09-06T11:02:54.256 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-06T11:02:54.258 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-09-06T11:02:54.279 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.091) 0:03:07.434 ****** 2021-09-06T11:02:54.286 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-09-06T11:02:54.288 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "a 2021-09-06T11:02:54.289 INFO:teuthology.task.ansible.out:nsible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-09-06T11:02:54.314 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.035) 0:03:07.470 ****** 2021-09-06T11:02:54.354 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-09-06T11:02:54.379 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.064) 0:03:07.534 ****** 2021-09-06T11:02:54.397 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-09-06T11:02:54.399 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-06T11:02:54.425 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.045) 0:03:07.580 ****** 2021-09-06T11:02:54.484 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-09-06T11:02:54.486 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-06T11:02:54.510 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.085) 0:03:07.665 ****** 2021-09-06T11:02:54.613 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-09-06T11:02:54.615 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "a 2021-09-06T11:02:54.617 INFO:teuthology.task.ansible.out:nsible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-06T11:02:54.642 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.131) 0:03:07.797 ****** 2021-09-06T11:02:54.669 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.027) 0:03:07.824 ****** 2021-09-06T11:02:54.752 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-09-06T11:02:54.753 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-06T11:02:54.775 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.106) 0:03:07.931 ****** 2021-09-06T11:02:54.866 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-06T11:02:54.867 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "name": "ubuntu", "state": "present", "system": false} 2021-09-06T11:02:54.891 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.115) 0:03:08.046 ****** 2021-09-06T11:02:54.961 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-09-06T11:02:54.963 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-06T11:02:54.985 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.093) 0:03:08.140 ****** 2021-09-06T11:02:55.005 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:54 +0000 (0:00:00.019) 0:03:08.160 ****** 2021-09-06T11:02:55.045 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-09-06T11:02:55.047 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:02:55.069 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:55 +0000 (0:00:00.064) 0:03:08.225 ****** 2021-09-06T11:02:55.233 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-06T11:02:55.235 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "name": "ubuntu", "state": "present", "system": false} 2021-09-06T11:02:55.257 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:55 +0000 (0:00:00.187) 0:03:08.412 ****** 2021-09-06T11:02:55.329 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-09-06T11:02:55.331 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi094.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-09-06T11:02:55.353 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:55 +0000 (0:00:00.095) 0:03:08.508 ****** 2021-09-06T11:02:55.373 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:55 +0000 (0:00:00.020) 0:03:08.528 ****** 2021-09-06T11:02:55.443 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-09-06T11:02:55.445 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:02:55.466 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:55 +0000 (0:00:00.092) 0:03:08.621 ****** 2021-09-06T11:02:55.723 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-09-06T11:02:55.726 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-09-06T11:02:55.746 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:55 +0000 (0:00:00.280) 0:03:08.902 ****** 2021-09-06T11:02:55.748 INFO:teuthology.task.ansible.out: 2021-09-06T11:02:55.923 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-09-06T11:02:55.926 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-09-06T11:02:55.952 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:55 +0000 (0:00:00.205) 0:03:09.107 ****** 2021-09-06T11:02:56.035 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-09-06T11:02:56.036 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi094"}, "changed": false} 2021-09-06T11:02:56.062 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:56 +0000 (0:00:00.110) 0:03:09.217 ****** 2021-09-06T11:02:56.344 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-09-06T11:02:56.346 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-09-06T11:02:56.368 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:56 +0000 (0:00:00.306) 0:03:09.524 ****** 2021-09-06T11:02:56.449 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-09-06T11:02:56.449 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi139"}, "changed": false} 2021-09-06T11:02:56.476 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:56 +0000 (0:00:00.107) 0:03:09.631 ****** 2021-09-06T11:02:57.044 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi094.front.sepia.ceph.com", "ansible_hostname": "smithi094", "ansible_nodename": "smithi094"}, "changed": true, "name": "smithi094"} 2021-09-06T11:02:57.069 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:57 +0000 (0:00:00.593) 0:03:10.224 ****** 2021-09-06T11:02:57.119 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi139.front.sepia.ceph.com", "ansible_hostname": "smithi139", "ansible_nodename": "smithi139"}, "changed": true, "name": "smithi139"} 2021-09-06T11:02:57.142 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:57 +0000 (0:00:00.073) 0:03:10.297 ****** 2021-09-06T11:02:57.561 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-09-06T11:02:57.562 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-09-06T11:02:57.586 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:57 +0000 (0:00:00.444) 0:03:10.742 ****** 2021-09-06T11:02:57.640 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-09-06T11:02:57.665 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:57 +0000 (0:00:00.078) 0:03:10.820 ****** 2021-09-06T11:02:58.079 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-09-06T11:02:58.081 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-09-06T11:02:58.103 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:58 +0000 (0:00:00.438) 0:03:11.259 ****** 2021-09-06T11:02:58.174 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-09-06T11:02:58.199 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:58 +0000 (0:00:00.095) 0:03:11.354 ****** 2021-09-06T11:02:58.798 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-09-06T11:02:58.800 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-06T11:02:58.822 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:58 +0000 (0:00:00.622) 0:03:11.977 ****** 2021-09-06T11:02:58.871 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-06T11:02:58.896 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:58 +0000 (0:00:00.074) 0:03:12.052 ****** 2021-09-06T11:02:59.345 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-09-06T11:02:59.347 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-09-06T11:02:59.369 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:59 +0000 (0:00:00.473) 0:03:12.525 ****** 2021-09-06T11:02:59.409 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-09-06T11:02:59.438 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:59 +0000 (0:00:00.067) 0:03:12.593 ****** 2021-09-06T11:02:59.604 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-06T11:02:59.606 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-06T11:02:59.629 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:59 +0000 (0:00:00.191) 0:03:12.784 ****** 2021-09-06T11:02:59.689 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-06T11:02:59.713 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:59 +0000 (0:00:00.083) 0:03:12.868 ****** 2021-09-06T11:02:59.856 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-06T11:02:59.858 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-09-06T11:02:59.880 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:59 +0000 (0:00:00.167) 0:03:13.035 ****** 2021-09-06T11:02:59.939 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-09-06T11:02:59.963 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:02:59 +0000 (0:00:00.082) 0:03:13.118 ****** 2021-09-06T11:03:00.150 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-06T11:03:00.152 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1254, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-06T11:03:00.174 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:00 +0000 (0:00:00.210) 0:03:13.329 ****** 2021-09-06T11:03:00.233 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1254, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-06T11:03:00.256 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:00 +0000 (0:00:00.082) 0:03:13.411 ****** 2021-09-06T11:03:00.684 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-09-06T11:03:00.706 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:00 +0000 (0:00:00.449) 0:03:13.861 ****** 2021-09-06T11:03:00.756 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-09-06T11:03:00.782 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:00 +0000 (0:00:00.076) 0:03:13.938 ****** 2021-09-06T11:03:01.209 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-06T11:03:01.211 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-09-06T11:03:01.232 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:01 +0000 (0:00:00.449) 0:03:14.387 ****** 2021-09-06T11:03:01.298 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-09-06T11:03:01.329 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:01 +0000 (0:00:00.096) 0:03:14.484 ****** 2021-09-06T11:03:03.832 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-06T11:03:03.834 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.366634", "end": "2021-09-06 11:03:03.808486", "rc": 0, "start": "2021-09-06 11:03:01.441852", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:03.855 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:03 +0000 (0:00:02.526) 0:03:17.011 ****** 2021-09-06T11:03:03.895 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.337618", "end": "2021-09-06 11:03:03.871201", "rc": 0, "start": "2021-09-06 11:03:01.533583", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:03.919 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:03 +0000 (0:00:00.063) 0:03:17.075 ****** 2021-09-06T11:03:03.937 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-09-06T11:03:03.939 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:03.969 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:03 +0000 (0:00:00.048) 0:03:17.124 ****** 2021-09-06T11:03:03.990 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:04.026 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:04 +0000 (0:00:00.056) 0:03:17.180 ****** 2021-09-06T11:03:04.045 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:04.047 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:04.070 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:04 +0000 (0:00:00.045) 0:03:17.225 ****** 2021-09-06T11:03:04.098 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:04.123 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:04 +0000 (0:00:00.053) 0:03:17.278 ****** 2021-09-06T11:03:04.151 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-09-06T11:03:04.153 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:04.178 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:04 +0000 (0:00:00.054) 0:03:17.333 ****** 2021-09-06T11:03:04.197 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:04.222 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:04 +0000 (0:00:00.043) 0:03:17.377 ****** 2021-09-06T11:03:04.696 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-09-06T11:03:04.698 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-09-06T11:03:04.751 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-09-06T11:03:05.123 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-09-06T11:03:05.189 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-09-06T11:03:05.546 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-09-06T11:03:05.610 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-09-06T11:03:05.979 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "20107eac0ba9a84b318fa5f29b02a387051bff98", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-09-06T11:03:06.017 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:06 +0000 (0:00:01.795) 0:03:19.173 ****** 2021-09-06T11:03:06.049 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "20107eac0ba9a84b318fa5f29b02a387051bff98", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-09-06T11:03:06.082 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:06 +0000 (0:00:00.064) 0:03:19.237 ****** 2021-09-06T11:03:06.539 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-09-06T11:03:06.541 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-09-06T11:03:06.564 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:06 +0000 (0:00:00.482) 0:03:19.719 ****** 2021-09-06T11:03:06.593 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-09-06T11:03:06.618 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:06 +0000 (0:00:00.053) 0:03:19.773 ****** 2021-09-06T11:03:07.077 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-09-06T11:03:07.080 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-09-06T11:03:07.134 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-09-06T11:03:07.521 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-09-06T11:03:07.565 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-09-06T11:03:07.942 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-09-06T11:03:07.995 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-09-06T11:03:08.370 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "state": "file", "uid": 0} 2021-09-06T11:03:08.396 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:08 +0000 (0:00:01.778) 0:03:21.552 ****** 2021-09-06T11:03:08.425 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "state": "file", "uid": 0} 2021-09-06T11:03:08.453 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:08 +0000 (0:00:00.056) 0:03:21.608 ****** 2021-09-06T11:03:08.954 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-09-06T11:03:08.956 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.320703", "end": "2021-09-06 11:03:08.933342", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-09-06 11:03:08.612639", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'comma 2021-09-06T11:03:08.957 INFO:teuthology.task.ansible.out:nd_warnings=False' in ansible.cfg to get rid of this message. 2021-09-06T11:03:08.978 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:08 +0000 (0:00:00.525) 0:03:22.134 ****** 2021-09-06T11:03:09.007 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.313856", "end": "2021-09-06 11:03:08.980808", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-09-06 11:03:08.666952", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-09-06T11:03:09.033 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.054) 0:03:22.188 ****** 2021-09-06T11:03:09.050 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-09-06T11:03:09.052 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.077 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.043) 0:03:22.232 ****** 2021-09-06T11:03:09.106 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.130 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.053) 0:03:22.285 ****** 2021-09-06T11:03:09.159 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-09-06T11:03:09.160 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.162 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.185 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.054) 0:03:22.340 ****** 2021-09-06T11:03:09.213 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.224 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.249 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.064) 0:03:22.404 ****** 2021-09-06T11:03:09.268 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-09-06T11:03:09.271 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.295 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.045) 0:03:22.450 ****** 2021-09-06T11:03:09.345 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.370 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.075) 0:03:22.526 ****** 2021-09-06T11:03:09.378 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:09.380 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-09-06T11:03:09.402 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.031) 0:03:22.558 ****** 2021-09-06T11:03:09.442 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-09-06T11:03:09.469 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.066) 0:03:22.624 ****** 2021-09-06T11:03:09.476 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-09-06T11:03:09.478 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.500 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.031) 0:03:22.656 ****** 2021-09-06T11:03:09.540 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.567 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.064) 0:03:22.720 ****** 2021-09-06T11:03:09.573 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.598 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.032) 0:03:22.753 ****** 2021-09-06T11:03:09.636 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.660 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.062) 0:03:22.815 ****** 2021-09-06T11:03:09.838 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-09-06T11:03:09.840 INFO:teuthology.task.ansible.out:fatal: [smithi094.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006595", "end": "2021-09-06 11:03:09.813066", "msg": "non-zero return code", "rc": 1, "start": "2021-09-06 11:03:09.806471", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-09-06T11:03:09.862 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.201) 0:03:23.017 ****** 2021-09-06T11:03:09.913 INFO:teuthology.task.ansible.out:fatal: [smithi139.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006575", "end": "2021-09-06 11:03:09.885961", "msg": "non-zero return code", "rc": 1, "start": "2021-09-06 11:03:09.879386", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-09-06T11:03:09.938 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.075) 0:03:23.093 ****** 2021-09-06T11:03:09.955 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-09-06T11:03:09.956 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:09.978 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:09 +0000 (0:00:00.040) 0:03:23.134 ****** 2021-09-06T11:03:10.028 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:10.054 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:10 +0000 (0:00:00.073) 0:03:23.207 ****** 2021-09-06T11:03:14.628 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-06T11:03:14.630 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2021-09-06T11:03:14.651 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:14 +0000 (0:00:04.598) 0:03:27.806 ****** 2021-09-06T11:03:14.710 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2021-09-06T11:03:14.735 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:14 +0000 (0:00:00.083) 0:03:27.890 ****** 2021-09-06T11:03:17.219 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-09-06T11:03:17.241 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:17 +0000 (0:00:02.506) 0:03:30.396 ****** 2021-09-06T11:03:17.323 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-09-06T11:03:17.347 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:17 +0000 (0:00:00.106) 0:03:30.502 ****** 2021-09-06T11:03:24.422 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T11:03:24.424 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2021-09-06T11:03:24.445 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:24 +0000 (0:00:07.098) 0:03:37.600 ****** 2021-09-06T11:03:24.526 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-09-06T11:03:24.527 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:24.548 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:24 +0000 (0:00:00.102) 0:03:37.703 ****** 2021-09-06T11:03:25.064 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T11:03:25.066 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2021-09-06T11:03:25.087 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:25 +0000 (0:00:00.539) 0:03:38.243 ****** 2021-09-06T11:03:25.156 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-09-06T11:03:25.158 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:25.178 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:25 +0000 (0:00:00.091) 0:03:38.334 ****** 2021-09-06T11:03:26.495 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-06T11:03:26.497 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:03:26.519 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:26 +0000 (0:00:01.340) 0:03:39.674 ****** 2021-09-06T11:03:27.141 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:03:27.166 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:27 +0000 (0:00:00.647) 0:03:40.321 ****** 2021-09-06T11:03:28.390 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T11:03:28.392 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:03:28.414 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:28 +0000 (0:00:01.248) 0:03:41.570 ****** 2021-09-06T11:03:28.801 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-09-06T11:03:28.803 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveState": "inactive", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity 2021-09-06T11:03:28.805 INFO:teuthology.task.ansible.out:", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-06T11:03:28.836 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:28 +0000 (0:00:00.421) 0:03:41.991 ****** 2021-09-06T11:03:29.156 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T11:03:29.158 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:03:29.179 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:29 +0000 (0:00:00.343) 0:03:42.335 ****** 2021-09-06T11:03:29.218 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-09-06T11:03:29.219 INFO:teuthology.task.ansible.out:fatal: [smithi094.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-09-06T11:03:29.245 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:29 +0000 (0:00:00.065) 0:03:42.400 ****** 2021-09-06T11:03:29.557 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-09-06T11:03:29.559 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveState": "inactive", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity 2021-09-06T11:03:29.560 INFO:teuthology.task.ansible.out:", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-06T11:03:29.581 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:29 +0000 (0:00:00.336) 0:03:42.737 ****** 2021-09-06T11:03:29.881 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-09-06T11:03:29.903 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:29 +0000 (0:00:00.322) 0:03:43.059 ****** 2021-09-06T11:03:29.963 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-09-06T11:03:29.965 INFO:teuthology.task.ansible.out:fatal: [smithi139.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-09-06T11:03:29.988 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:29 +0000 (0:00:00.084) 0:03:43.143 ****** 2021-09-06T11:03:29.996 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-09-06T11:03:29.998 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.022 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.033) 0:03:43.177 ****** 2021-09-06T11:03:30.093 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-09-06T11:03:30.095 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.115 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.093) 0:03:43.271 ****** 2021-09-06T11:03:30.185 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.208 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.092) 0:03:43.363 ****** 2021-09-06T11:03:30.277 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.300 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.092) 0:03:43.455 ****** 2021-09-06T11:03:30.328 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-09-06T11:03:30.352 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.051) 0:03:43.507 ****** 2021-09-06T11:03:30.370 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-09-06T11:03:30.372 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.394 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.041) 0:03:43.549 ****** 2021-09-06T11:03:30.422 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-09-06T11:03:30.424 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.446 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.051) 0:03:43.601 ****** 2021-09-06T11:03:30.464 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-09-06T11:03:30.466 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.487 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.041) 0:03:43.642 ****** 2021-09-06T11:03:30.515 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-09-06T11:03:30.517 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.540 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.052) 0:03:43.695 ****** 2021-09-06T11:03:30.557 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-09-06T11:03:30.580 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.040) 0:03:43.736 ****** 2021-09-06T11:03:30.608 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-09-06T11:03:30.611 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.633 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.052) 0:03:43.788 ****** 2021-09-06T11:03:30.650 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-09-06T11:03:30.652 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.674 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.041) 0:03:43.829 ****** 2021-09-06T11:03:30.713 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-09-06T11:03:30.715 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.737 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.063) 0:03:43.892 ****** 2021-09-06T11:03:30.746 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-09-06T11:03:30.748 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.779 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.041) 0:03:43.934 ****** 2021-09-06T11:03:30.822 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-09-06T11:03:30.823 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.846 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.067) 0:03:44.001 ****** 2021-09-06T11:03:30.863 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-09-06T11:03:30.865 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.889 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.042) 0:03:44.044 ****** 2021-09-06T11:03:30.916 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-09-06T11:03:30.918 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.939 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.050) 0:03:44.094 ****** 2021-09-06T11:03:30.956 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-09-06T11:03:30.958 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:30.978 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:30 +0000 (0:00:00.039) 0:03:44.133 ****** 2021-09-06T11:03:31.006 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-09-06T11:03:31.029 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.050) 0:03:44.184 ****** 2021-09-06T11:03:31.046 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-09-06T11:03:31.069 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.039) 0:03:44.224 ****** 2021-09-06T11:03:31.098 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.125 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.055) 0:03:44.280 ****** 2021-09-06T11:03:31.142 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-09-06T11:03:31.144 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.167 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.042) 0:03:44.323 ****** 2021-09-06T11:03:31.186 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-09-06T11:03:31.187 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.211 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.043) 0:03:44.366 ****** 2021-09-06T11:03:31.240 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-09-06T11:03:31.241 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.263 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.052) 0:03:44.419 ****** 2021-09-06T11:03:31.281 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-09-06T11:03:31.282 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.306 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.042) 0:03:44.461 ****** 2021-09-06T11:03:31.334 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-09-06T11:03:31.335 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.357 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.051) 0:03:44.512 ****** 2021-09-06T11:03:31.375 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-09-06T11:03:31.376 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.398 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.041) 0:03:44.554 ****** 2021-09-06T11:03:31.427 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T11:03:31.429 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.451 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.051) 0:03:44.606 ****** 2021-09-06T11:03:31.468 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-09-06T11:03:31.492 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.041) 0:03:44.647 ****** 2021-09-06T11:03:31.521 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-09-06T11:03:31.522 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.550 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.058) 0:03:44.705 ****** 2021-09-06T11:03:31.569 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-09-06T11:03:31.570 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.593 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.042) 0:03:44.748 ****** 2021-09-06T11:03:31.621 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T11:03:31.623 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.645 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.051) 0:03:44.800 ****** 2021-09-06T11:03:31.663 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-09-06T11:03:31.665 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.688 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.043) 0:03:44.844 ****** 2021-09-06T11:03:31.718 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.741 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.052) 0:03:44.896 ****** 2021-09-06T11:03:31.757 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-09-06T11:03:31.759 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.781 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.040) 0:03:44.937 ****** 2021-09-06T11:03:31.821 INFO:teuthology.task.ansible.out: 2021-09-06T11:03:31.824 INFO:teuthology.task.ansible.out:TASK [testnode : Install packages with --no-install-recommends] **************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.847 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.065) 0:03:45.002 ****** 2021-09-06T11:03:31.927 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-09-06T11:03:31.929 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:31.950 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:31 +0000 (0:00:00.102) 0:03:45.105 ****** 2021-09-06T11:03:32.009 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T11:03:32.010 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.031 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.081) 0:03:45.186 ****** 2021-09-06T11:03:32.038 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-09-06T11:03:32.039 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.062 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.030) 0:03:45.217 ****** 2021-09-06T11:03:32.100 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-09-06T11:03:32.101 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.124 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.062) 0:03:45.279 ****** 2021-09-06T11:03:32.131 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-09-06T11:03:32.133 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.156 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.032) 0:03:45.311 ****** 2021-09-06T11:03:32.195 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T11:03:32.197 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.221 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.064) 0:03:45.376 ****** 2021-09-06T11:03:32.228 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-09-06T11:03:32.230 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.253 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.031) 0:03:45.408 ****** 2021-09-06T11:03:32.291 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-09-06T11:03:32.293 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.319 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.066) 0:03:45.474 ****** 2021-09-06T11:03:32.325 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-09-06T11:03:32.326 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.349 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.029) 0:03:45.504 ****** 2021-09-06T11:03:32.377 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-09-06T11:03:32.378 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.401 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.051) 0:03:45.556 ****** 2021-09-06T11:03:32.418 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-09-06T11:03:32.420 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.441 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.040) 0:03:45.597 ****** 2021-09-06T11:03:32.469 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-09-06T11:03:32.471 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.493 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.051) 0:03:45.648 ****** 2021-09-06T11:03:32.510 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-09-06T11:03:32.512 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.533 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.040) 0:03:45.689 ****** 2021-09-06T11:03:32.561 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-09-06T11:03:32.562 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.584 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.050) 0:03:45.740 ****** 2021-09-06T11:03:32.607 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-06T11:03:32.609 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.630 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.046) 0:03:45.786 ****** 2021-09-06T11:03:32.658 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-09-06T11:03:32.660 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.682 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.051) 0:03:45.838 ****** 2021-09-06T11:03:32.701 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-06T11:03:32.703 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.725 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.042) 0:03:45.880 ****** 2021-09-06T11:03:32.744 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-09-06T11:03:32.746 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.768 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.043) 0:03:45.924 ****** 2021-09-06T11:03:32.786 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.811 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.042) 0:03:45.966 ****** 2021-09-06T11:03:32.849 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-09-06T11:03:32.851 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.874 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.063) 0:03:46.030 ****** 2021-09-06T11:03:32.892 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-06T11:03:32.894 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.923 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.048) 0:03:46.079 ****** 2021-09-06T11:03:32.943 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-09-06T11:03:32.945 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:32.967 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:32 +0000 (0:00:00.043) 0:03:46.122 ****** 2021-09-06T11:03:32.995 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-06T11:03:32.997 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.019 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.051) 0:03:46.174 ****** 2021-09-06T11:03:33.196 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-09-06T11:03:33.198 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.219 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.200) 0:03:46.374 ****** 2021-09-06T11:03:33.236 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-06T11:03:33.238 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.262 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.042) 0:03:46.417 ****** 2021-09-06T11:03:33.439 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.461 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.199) 0:03:46.616 ****** 2021-09-06T11:03:33.479 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-06T11:03:33.481 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.502 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.041) 0:03:46.657 ****** 2021-09-06T11:03:33.573 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-06T11:03:33.575 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.595 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.093) 0:03:46.751 ****** 2021-09-06T11:03:33.665 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:33.667 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.691 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.095) 0:03:46.847 ****** 2021-09-06T11:03:33.698 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-06T11:03:33.700 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.724 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.032) 0:03:46.879 ****** 2021-09-06T11:03:33.761 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-06T11:03:33.763 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.786 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.061) 0:03:46.941 ****** 2021-09-06T11:03:33.792 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-06T11:03:33.794 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.816 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.030) 0:03:46.971 ****** 2021-09-06T11:03:33.843 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T11:03:33.845 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.868 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.051) 0:03:47.023 ****** 2021-09-06T11:03:33.886 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-06T11:03:33.888 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.910 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.041) 0:03:47.065 ****** 2021-09-06T11:03:33.928 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T11:03:33.929 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:33.952 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.042) 0:03:47.108 ****** 2021-09-06T11:03:33.980 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-06T11:03:33.982 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:34.004 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:33 +0000 (0:00:00.052) 0:03:47.160 ****** 2021-09-06T11:03:34.021 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-06T11:03:34.023 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:34.046 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:34 +0000 (0:00:00.041) 0:03:47.201 ****** 2021-09-06T11:03:34.075 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-06T11:03:34.077 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:34.099 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:34 +0000 (0:00:00.053) 0:03:47.255 ****** 2021-09-06T11:03:34.162 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:34.184 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:34 +0000 (0:00:00.084) 0:03:47.339 ****** 2021-09-06T11:03:34.254 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:34.276 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:34 +0000 (0:00:00.092) 0:03:47.431 ****** 2021-09-06T11:03:34.293 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T11:03:34.295 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-09-06T11:03:34.320 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:34 +0000 (0:00:00.043) 0:03:47.475 ****** 2021-09-06T11:03:34.350 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:34.373 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:34 +0000 (0:00:00.053) 0:03:47.528 ****** 2021-09-06T11:03:34.593 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-06T11:03:34.594 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:34.615 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:34 +0000 (0:00:00.241) 0:03:47.770 ****** 2021-09-06T11:03:34.748 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T11:03:34.750 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T11:03:34.774 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:34 +0000 (0:00:00.158) 0:03:47.929 ****** 2021-09-06T11:03:34.824 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T11:03:34.826 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:34.847 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:34 +0000 (0:00:00.073) 0:03:48.002 ****** 2021-09-06T11:03:35.056 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T11:03:35.058 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:35.081 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:35 +0000 (0:00:00.234) 0:03:48.236 ****** 2021-09-06T11:03:35.194 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-06T11:03:35.196 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-09-06T11:03:35.350 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:35 +0000 (0:00:00.268) 0:03:48.505 ****** 2021-09-06T11:03:35.357 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-06T11:03:35.359 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:35.382 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:35 +0000 (0:00:00.032) 0:03:48.537 ****** 2021-09-06T11:03:35.420 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-06T11:03:35.422 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:35.446 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:35 +0000 (0:00:00.063) 0:03:48.601 ****** 2021-09-06T11:03:35.516 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-06T11:03:35.518 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:35.544 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:35 +0000 (0:00:00.098) 0:03:48.699 ****** 2021-09-06T11:03:35.614 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-06T11:03:35.616 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:35.640 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:35 +0000 (0:00:00.095) 0:03:48.795 ****** 2021-09-06T11:03:35.648 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T11:03:35.650 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-09-06T11:03:35.677 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:35 +0000 (0:00:00.037) 0:03:48.832 ****** 2021-09-06T11:03:35.706 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:35.730 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:35 +0000 (0:00:00.052) 0:03:48.885 ****** 2021-09-06T11:03:35.799 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T11:03:35.802 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:35.822 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:35 +0000 (0:00:00.092) 0:03:48.977 ****** 2021-09-06T11:03:35.892 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:35.922 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:35 +0000 (0:00:00.099) 0:03:49.077 ****** 2021-09-06T11:03:35.994 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T11:03:35.995 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.016 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.094) 0:03:49.172 ****** 2021-09-06T11:03:36.025 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T11:03:36.027 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T11:03:36.050 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.033) 0:03:49.205 ****** 2021-09-06T11:03:36.089 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T11:03:36.091 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.114 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.062) 0:03:49.268 ****** 2021-09-06T11:03:36.184 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-06T11:03:36.185 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.206 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.093) 0:03:49.361 ****** 2021-09-06T11:03:36.275 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-09-06T11:03:36.277 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.298 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.092) 0:03:49.453 ****** 2021-09-06T11:03:36.306 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-06T11:03:36.306 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-09-06T11:03:36.331 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.032) 0:03:49.486 ****** 2021-09-06T11:03:36.370 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-09-06T11:03:36.372 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.393 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.062) 0:03:49.549 ****** 2021-09-06T11:03:36.412 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-06T11:03:36.413 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.437 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.043) 0:03:49.593 ****** 2021-09-06T11:03:36.465 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-06T11:03:36.467 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.492 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.054) 0:03:49.647 ****** 2021-09-06T11:03:36.509 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-06T11:03:36.511 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.533 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.041) 0:03:49.688 ****** 2021-09-06T11:03:36.561 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-06T11:03:36.563 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.585 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.051) 0:03:49.740 ****** 2021-09-06T11:03:36.603 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-06T11:03:36.605 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.627 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.042) 0:03:49.783 ****** 2021-09-06T11:03:36.656 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-09-06T11:03:36.657 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.680 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.052) 0:03:49.835 ****** 2021-09-06T11:03:36.698 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T11:03:36.699 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.721 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.041) 0:03:49.876 ****** 2021-09-06T11:03:36.748 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-09-06T11:03:36.749 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.772 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.051) 0:03:49.928 ****** 2021-09-06T11:03:36.791 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T11:03:36.793 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.815 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.042) 0:03:49.970 ****** 2021-09-06T11:03:36.842 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-09-06T11:03:36.844 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.866 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.050) 0:03:50.021 ****** 2021-09-06T11:03:36.884 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.907 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.041) 0:03:50.063 ****** 2021-09-06T11:03:36.935 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-09-06T11:03:36.937 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:36.959 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.051) 0:03:50.114 ****** 2021-09-06T11:03:36.977 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T11:03:36.978 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.001 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:36 +0000 (0:00:00.042) 0:03:50.156 ****** 2021-09-06T11:03:37.029 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-09-06T11:03:37.030 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.054 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.053) 0:03:50.210 ****** 2021-09-06T11:03:37.074 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T11:03:37.076 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.098 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.043) 0:03:50.253 ****** 2021-09-06T11:03:37.125 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-09-06T11:03:37.127 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.149 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.050) 0:03:50.304 ****** 2021-09-06T11:03:37.166 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-06T11:03:37.168 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.189 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.040) 0:03:50.345 ****** 2021-09-06T11:03:37.217 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-09-06T11:03:37.218 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.240 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.050) 0:03:50.395 ****** 2021-09-06T11:03:37.257 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-09-06T11:03:37.259 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.281 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.040) 0:03:50.436 ****** 2021-09-06T11:03:37.308 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-09-06T11:03:37.310 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.332 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.050) 0:03:50.487 ****** 2021-09-06T11:03:37.349 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-09-06T11:03:37.351 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.374 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.042) 0:03:50.529 ****** 2021-09-06T11:03:37.401 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-09-06T11:03:37.403 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.425 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.050) 0:03:50.580 ****** 2021-09-06T11:03:37.442 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-06T11:03:37.444 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.468 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.043) 0:03:50.623 ****** 2021-09-06T11:03:37.506 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:37.508 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-06T11:03:37.530 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.062) 0:03:50.685 ****** 2021-09-06T11:03:37.552 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-06T11:03:37.553 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.576 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.046) 0:03:50.732 ****** 2021-09-06T11:03:37.624 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-09-06T11:03:37.627 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.636 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-06T11:03:37.646 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.648 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi094.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.674 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.097) 0:03:50.829 ****** 2021-09-06T11:03:37.681 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.682 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.712 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.038) 0:03:50.867 ****** 2021-09-06T11:03:37.740 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-09-06T11:03:37.742 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.767 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.055) 0:03:50.923 ****** 2021-09-06T11:03:37.839 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-09-06T11:03:37.841 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.864 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.096) 0:03:51.019 ****** 2021-09-06T11:03:37.934 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-09-06T11:03:37.936 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:37.956 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.091) 0:03:51.111 ****** 2021-09-06T11:03:37.962 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-09-06T11:03:37.964 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003214", "end": "2021-09-06 11:03:37.930010", "rc": 0, "start": "2021-09-06 11:03:37.926796", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock"]} 2021-09-06T11:03:37.990 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:37 +0000 (0:00:00.033) 0:03:51.145 ****** 2021-09-06T11:03:38.028 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-09-06T11:03:38.030 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.052 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.062) 0:03:51.207 ****** 2021-09-06T11:03:38.070 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-09-06T11:03:38.072 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.097 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.044) 0:03:51.252 ****** 2021-09-06T11:03:38.125 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-09-06T11:03:38.126 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.149 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.052) 0:03:51.304 ****** 2021-09-06T11:03:38.167 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-09-06T11:03:38.169 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.192 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.043) 0:03:51.348 ****** 2021-09-06T11:03:38.223 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.247 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.054) 0:03:51.402 ****** 2021-09-06T11:03:38.264 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-09-06T11:03:38.266 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.288 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.041) 0:03:51.443 ****** 2021-09-06T11:03:38.326 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-09-06T11:03:38.328 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.349 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.061) 0:03:51.505 ****** 2021-09-06T11:03:38.367 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-09-06T11:03:38.369 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.391 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.041) 0:03:51.546 ****** 2021-09-06T11:03:38.419 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-09-06T11:03:38.422 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.444 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.052) 0:03:51.599 ****** 2021-09-06T11:03:38.524 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:38.526 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-06T11:03:38.547 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.103) 0:03:51.702 ****** 2021-09-06T11:03:38.639 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-09-06T11:03:38.641 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.651 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-06T11:03:38.653 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.662 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.663 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.674 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.675 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-09-06T11:03:38.701 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.153) 0:03:51.856 ****** 2021-09-06T11:03:38.947 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-09-06T11:03:38.949 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003728", "end": "2021-09-06 11:03:38.919967", "rc": 0, "start": "2021-09-06 11:03:38.916239", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock"]} 2021-09-06T11:03:38.973 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:38 +0000 (0:00:00.272) 0:03:52.129 ****** 2021-09-06T11:03:39.053 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:39.084 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:39 +0000 (0:00:00.110) 0:03:52.239 ****** 2021-09-06T11:03:39.153 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-09-06T11:03:39.155 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:39.175 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:39 +0000 (0:00:00.091) 0:03:52.331 ****** 2021-09-06T11:03:39.244 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:39.267 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:39 +0000 (0:00:00.091) 0:03:52.422 ****** 2021-09-06T11:03:39.336 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-09-06T11:03:39.338 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:39.358 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:39 +0000 (0:00:00.091) 0:03:52.513 ****** 2021-09-06T11:03:40.017 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-09-06T11:03:40.019 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:03:40.042 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:40 +0000 (0:00:00.683) 0:03:53.197 ****** 2021-09-06T11:03:40.133 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-06T11:03:40.135 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930261512192, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114627, 'block_used': 12980481, 'inode_total': 61054976, 'inode_available': 60986042, 'inode_used': 68934, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114627, "block_size": 4096, "block_total": 240095108, "block_used": 12980481, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986042, "inode_total": 61054976, "inode_used": 68934, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930261512192, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}} 2021-09-06T11:03:40.159 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:40 +0000 (0:00:00.116) 0:03:53.314 ****** 2021-09-06T11:03:40.398 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005167", "end": "2021-09-06 11:03:40.374396", "rc": 0, "start": "2021-09-06 11:03:40.369229", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:40.420 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:40 +0000 (0:00:00.261) 0:03:53.575 ****** 2021-09-06T11:03:40.501 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-09-06T11:03:40.502 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930261512192, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114627, 'block_used': 12980481, 'inode_total': 61054976, 'inode_available': 60986042, 'inode_used': 68934, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114627, "block_size": 4096, "block_total": 240095108, "block_used": 12980481, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986042, "inode_total": 61054976, "inode_used": 68934, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930261512192, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:40.524 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:40 +0000 (0:00:00.104) 0:03:53.680 ****** 2021-09-06T11:03:40.596 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-09-06T11:03:40.619 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:40 +0000 (0:00:00.094) 0:03:53.774 ****** 2021-09-06T11:03:40.955 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-09-06T11:03:40.958 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:03:40.978 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:40 +0000 (0:00:00.359) 0:03:54.133 ****** 2021-09-06T11:03:41.067 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-06T11:03:41.069 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930261520384, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114629, 'block_used': 12980479, 'inode_total': 61054976, 'inode_available': 60986042, 'inode_used': 68934, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114629, "block_size": 4096, "block_total": 240095108, "block_used": 12980479, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986042, "inode_total": 61054976, "inode_used": 68934, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930261520384, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}} 2021-09-06T11:03:41.090 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:41 +0000 (0:00:00.112) 0:03:54.246 ****** 2021-09-06T11:03:41.328 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-09-06T11:03:41.330 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005121", "end": "2021-09-06 11:03:41.304088", "rc": 0, "start": "2021-09-06 11:03:41.298967", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:41.350 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:41 +0000 (0:00:00.259) 0:03:54.505 ****** 2021-09-06T11:03:41.429 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi139.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930261520384, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114629, 'block_used': 12980479, 'inode_total': 61054976, 'inode_available': 60986042, 'inode_used': 68934, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114629, "block_size": 4096, "block_total": 240095108, "block_used": 12980479, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986042, "inode_total": 61054976, "inode_used": 68934, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930261520384, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:41.451 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:41 +0000 (0:00:00.101) 0:03:54.607 ****** 2021-09-06T11:03:41.521 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-09-06T11:03:41.551 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:41 +0000 (0:00:00.099) 0:03:54.706 ****** 2021-09-06T11:03:42.437 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-09-06T11:03:42.439 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDucfSRlI2Bo3Z1sRp3ulc2ySKfoOo7yLP'], 'uuids': ['aebd6140-3012-4620-9a98-fa1d95f5ca4a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.550050", "end": "2021-09-06 11:03:42.402742", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDucfSRlI2Bo3Z1sRp3ulc2ySKfoOo7yLP"], "labels": [], "masters": [], "uuids": ["aebd6140-3012-4620-9a98-fa1d95f5ca4a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:40.852692", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:43.147 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpP8qCthRGhVzrIBsA85TxXLtlxh8MAiOb'], 'uuids': ['c6d06871-bde9-4cfd-be0d-b1480f1478ca'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.335782", "end": "2021-09-06 11:03:43.114567", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpP8qCthRGhVzrIBsA85TxXLtlxh8MAiOb"], "labels": [], "masters": [], "uuids": ["c6d06871-bde9-4cfd-be0d-b1480f1478ca"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:41.778785", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:43.689 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-MqdVyc-YK8a-nObU-duWd-lyJO-aW5N-Z8zB3e', 'nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN', 'nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.069911", "end": "2021-09-06 11:03:43.657959", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-MqdVyc-YK8a-nObU-duWd-lyJO-aW5N-Z8zB3e", "nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN", "nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:42.588048", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-09-06T11:03:43.690 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:44.417 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-fAwtgW-KZMm-VbUR-qQ92-NM4l-DwYF-00wzyv', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN', 'nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.086505", "end": "2021-09-06 11:03:44.390543", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-fAwtgW-KZMm-VbUR-qQ92-NM4l-DwYF-00wzyv", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN", "nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:43.304038", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-09-06T11:03:44.419 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:44.911 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSD7WfbBCymG9SWVerdcBBi2RcOforvcaEQ'], 'uuids': ['8fb548de-38ea-49a2-9ace-819fef39412a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.042727", "end": "2021-09-06 11:03:44.886634", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSD7WfbBCymG9SWVerdcBBi2RcOforvcaEQ"], "labels": [], "masters": [], "uuids": ["8fb548de-38ea-49a2-9ace-819fef39412a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:43.843907", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:45.674 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpx3v1Is91ReQgqhDzXaG5vrNF0QuBpfDR'], 'uuids': ['11beb121-0c3b-4aa0-aabb-e827ad78298d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.061116", "end": "2021-09-06 11:03:45.636630", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpx3v1Is91ReQgqhDzXaG5vrNF0QuBpfDR"], "labels": [], "masters": [], "uuids": ["11beb121-0c3b-4aa0-aabb-e827ad78298d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:44.575514", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:46.154 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDZUeObFj7ERrgvQFfFqK3zSHDsC7j3FPb'], 'uuids': ['e9721470-adca-424a-8251-288815934cd5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.058065", "end": "2021-09-06 11:03:46.128843", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDZUeObFj7ERrgvQFfFqK3zSHDsC7j3FPb"], "labels": [], "masters": [], "uuids": ["e9721470-adca-424a-8251-288815934cd5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:45.070778", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:46.913 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpHGfnVcNX1aIEpQE5ms1pPOqpgpo8vxA5'], 'uuids': ['10151b79-e67b-4299-9842-e740c664a909'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.064930", "end": "2021-09-06 11:03:46.886225", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpHGfnVcNX1aIEpQE5ms1pPOqpgpo8vxA5"], "labels": [], "masters": [], "uuids": ["10151b79-e67b-4299-9842-e740c664a909"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:45.821295", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:47.413 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDTAgD9H38y0KdUPsccWeQJGQRfDDSborA'], 'uuids': ['26c726fa-f3cc-4cd5-b1e9-50f3a09b4ee6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.069311", "end": "2021-09-06 11:03:47.386864", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDTAgD9H38y0KdUPsccWeQJGQRfDDSborA"], "labels": [], "masters": [], "uuids": ["26c726fa-f3cc-4cd5-b1e9-50f3a09b4ee6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:46.317553", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:47.425 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJNP', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJNP', 'scsi-35000c500918dbfac', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LJNP', 'wwn-0x5000c500918dbfac'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5LJNP', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500918dbfac', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJNP-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJNP-part1', 'scsi-35000c500918dbfac-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LJNP-part1', 'wwn-0x5000c500918dbfac-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJNP", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJNP", "scsi-35000c500918dbfac", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LJNP", "wwn-0x5000c500918dbfac"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJNP-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-06T11:03:47.427 INFO:teuthology.task.ansible.out:Z1W5LJNP-part1", "scsi-35000c500918dbfac-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LJNP-part1", "wwn-0x5000c500918dbfac-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJNP", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918dbfac"}}, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:48.165 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpmi7yNxDUAygobP2iMMkjqfeCgoffaKYY'], 'uuids': ['a0115370-5e16-4729-9b1c-821712ef93e1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.069098", "end": "2021-09-06 11:03:48.137743", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpmi7yNxDUAygobP2iMMkjqfeCgoffaKYY"], "labels": [], "masters": [], "uuids": ["a0115370-5e16-4729-9b1c-821712ef93e1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:47.068645", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:48.177 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFR4', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4', 'scsi-35000c500929226ed', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4', 'wwn-0x5000c500929226ed'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFR4', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500929226ed', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4-part1', 'scsi-35000c500929226ed-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4-part1', 'wwn-0x5000c500929226ed-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4", "scsi-35000c500929226ed", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4", "wwn-0x5000c500929226ed"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-06T11:03:48.179 INFO:teuthology.task.ansible.out:Z1W5XFR4-part1", "scsi-35000c500929226ed-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4-part1", "wwn-0x5000c500929226ed-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFR4", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500929226ed"}}, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:48.692 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDrNPdrmUonQzG3zW2de5gcvn4oBSvkBJy'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.072739", "end": "2021-09-06 11:03:48.655210", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDrNPdrmUonQzG3zW2de5gcvn4oBSvkBJy"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:47.582471", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:48.722 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:48 +0000 (0:00:07.171) 0:04:01.877 ****** 2021-09-06T11:03:49.084 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T11:03:49.086 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDucfSRlI2Bo3Z1sRp3ulc2ySKfoOo7yLP'], 'uuids': ['aebd6140-3012-4620-9a98-fa1d95f5ca4a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.100419", "end": "2021-09-06 11:03:49.058699", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDucfSRlI2Bo3Z1sRp3ulc2ySKfoOo7yLP"], "labels": [], "masters": [], "uuids": ["aebd6140-3012-4620-9a98-fa1d95f5ca4a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:48.958280", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0976582 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0976582 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:49.303 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-MqdVyc-YK8a-nObU-duWd-lyJO-aW5N-Z8zB3e', 'nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN', 'nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039942", "end": "2021-09-06 11:03:49.272370", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-MqdVyc-YK8a-nObU-duWd-lyJO-aW5N-Z8zB3e", "nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN", "nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:49.232428", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.037306 s, 3.1 GB/s", "stderr_lines": ["1 2021-09-06T11:03:49.305 INFO:teuthology.task.ansible.out:10+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.037306 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:49.443 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-09-06T11:03:49.445 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBp5oQdMRHOzXBXtTNBSoWvGbGV4UcUWBAf'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.078611", "end": "2021-09-06 11:03:49.411514", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBp5oQdMRHOzXBXtTNBSoWvGbGV4UcUWBAf"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:48.332903", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T11:03:49.474 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:49 +0000 (0:00:00.751) 0:04:02.629 ****** 2021-09-06T11:03:49.576 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T11:03:49.578 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSD7WfbBCymG9SWVerdcBBi2RcOforvcaEQ'], 'uuids': ['8fb548de-38ea-49a2-9ace-819fef39412a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.099524", "end": "2021-09-06 11:03:49.546904", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSD7WfbBCymG9SWVerdcBBi2RcOforvcaEQ"], "labels": [], "masters": [], "uuids": ["8fb548de-38ea-49a2-9ace-819fef39412a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:49.447380", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0969197 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0969197 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:49.838 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDZUeObFj7ERrgvQFfFqK3zSHDsC7j3FPb'], 'uuids': ['e9721470-adca-424a-8251-288815934cd5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.098887", "end": "2021-09-06 11:03:49.818211", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDZUeObFj7ERrgvQFfFqK3zSHDsC7j3FPb"], "labels": [], "masters": [], "uuids": ["e9721470-adca-424a-8251-288815934cd5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:49.719324", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.096264 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.096264 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:49.902 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpP8qCthRGhVzrIBsA85TxXLtlxh8MAiOb'], 'uuids': ['c6d06871-bde9-4cfd-be0d-b1480f1478ca'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.149690", "end": "2021-09-06 11:03:49.875781", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpP8qCthRGhVzrIBsA85TxXLtlxh8MAiOb"], "labels": [], "masters": [], "uuids": ["c6d06871-bde9-4cfd-be0d-b1480f1478ca"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:49.726091", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146949 s, 785 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146949 s, 785 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:50.111 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDTAgD9H38y0KdUPsccWeQJGQRfDDSborA'], 'uuids': ['26c726fa-f3cc-4cd5-b1e9-50f3a09b4ee6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.097179", "end": "2021-09-06 11:03:50.089655", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDTAgD9H38y0KdUPsccWeQJGQRfDDSborA"], "labels": [], "masters": [], "uuids": ["26c726fa-f3cc-4cd5-b1e9-50f3a09b4ee6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:49.992476", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0946666 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0946666 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:50.112 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-fAwtgW-KZMm-VbUR-qQ92-NM4l-DwYF-00wzyv', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN', 'nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.040065", "end": "2021-09-06 11:03:50.088934", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-fAwtgW-KZMm-VbUR-qQ92-NM4l-DwYF-00wzyv", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN", "nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:50.048869", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0373351 s, 3.1 GB/s", "std 2021-09-06T11:03:50.114 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0373351 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:50.125 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJNP', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJNP', 'scsi-35000c500918dbfac', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LJNP', 'wwn-0x5000c500918dbfac'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5LJNP', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500918dbfac', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJNP-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJNP-part1', 'scsi-35000c500918dbfac-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LJNP-part1', 'wwn-0x5000c500918dbfac-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJNP", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJNP", "scsi-35000c500918dbfac", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LJNP", "wwn-0x5000c500918dbfac"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJNP-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-06T11:03:50.126 INFO:teuthology.task.ansible.out:Z1W5LJNP-part1", "scsi-35000c500918dbfac-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LJNP-part1", "wwn-0x5000c500918dbfac-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJNP", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918dbfac"}}, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:50.388 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDrNPdrmUonQzG3zW2de5gcvn4oBSvkBJy'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.099838", "end": "2021-09-06 11:03:50.365623", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-OB7cf22lGDh50QIMsgDRUYQ9nk2pVoSDrNPdrmUonQzG3zW2de5gcvn4oBSvkBJy"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:50.265785", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0973141 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0973141 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:50.417 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:50 +0000 (0:00:00.942) 0:04:03.572 ****** 2021-09-06T11:03:50.436 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpx3v1Is91ReQgqhDzXaG5vrNF0QuBpfDR'], 'uuids': ['11beb121-0c3b-4aa0-aabb-e827ad78298d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.141174", "end": "2021-09-06 11:03:50.404712", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpx3v1Is91ReQgqhDzXaG5vrNF0QuBpfDR"], "labels": [], "masters": [], "uuids": ["11beb121-0c3b-4aa0-aabb-e827ad78298d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:50.263538", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138479 s, 833 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138479 s, 833 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:50.710 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-09-06T11:03:50.712 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.061360", "end": "2021-09-06 11:03:50.642488", "rc": 0, "start": "2021-09-06 11:03:50.581128", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:50.732 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:50 +0000 (0:00:00.315) 0:04:03.888 ****** 2021-09-06T11:03:50.750 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T11:03:50.751 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpHGfnVcNX1aIEpQE5ms1pPOqpgpo8vxA5'], 'uuids': ['10151b79-e67b-4299-9842-e740c664a909'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.141072", "end": "2021-09-06 11:03:50.717610", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpHGfnVcNX1aIEpQE5ms1pPOqpgpo8vxA5"], "labels": [], "masters": [], "uuids": ["10151b79-e67b-4299-9842-e740c664a909"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:50.576538", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138436 s, 833 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138436 s, 833 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:50.962 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-09-06T11:03:50.964 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.042975", "end": "2021-09-06 11:03:50.939569", "rc": 0, "start": "2021-09-06 11:03:50.896594", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:50.986 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:50 +0000 (0:00:00.253) 0:04:04.141 ****** 2021-09-06T11:03:51.055 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-06T11:03:51.117 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.130) 0:04:04.272 ****** 2021-09-06T11:03:51.124 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T11:03:51.126 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpmi7yNxDUAygobP2iMMkjqfeCgoffaKYY'], 'uuids': ['a0115370-5e16-4729-9b1c-821712ef93e1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.145581", "end": "2021-09-06 11:03:51.038188", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBpmi7yNxDUAygobP2iMMkjqfeCgoffaKYY"], "labels": [], "masters": [], "uuids": ["a0115370-5e16-4729-9b1c-821712ef93e1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:50.892607", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142972 s, 807 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142972 s, 807 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:51.127 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFR4', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4', 'scsi-35000c500929226ed', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4', 'wwn-0x5000c500929226ed'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_ 2021-09-06T11:03:51.129 INFO:teuthology.task.ansible.out:address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFR4', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500929226ed', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4-part1', 'scsi-35000c500929226ed-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4-part1', 'wwn-0x5000c500929226ed-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4", "scsi-35000c500929226ed", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4", "wwn-0x5000c500929226ed"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4-part1", "scsi-35000c500929226ed-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4-part1", "wwn-0x5000c500929226ed-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_ha 2021-09-06T11:03:51.130 INFO:teuthology.task.ansible.out:ndle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFR4", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500929226ed"}}, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.139 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-09-06T11:03:51.141 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.164 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.047) 0:04:04.320 ****** 2021-09-06T11:03:51.192 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-09-06T11:03:51.194 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.215 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.050) 0:04:04.370 ****** 2021-09-06T11:03:51.243 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-06T11:03:51.244 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.268 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.053) 0:04:04.424 ****** 2021-09-06T11:03:51.296 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-09-06T11:03:51.298 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.319 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.050) 0:04:04.474 ****** 2021-09-06T11:03:51.392 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T11:03:51.394 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBp5oQdMRHOzXBXtTNBSoWvGbGV4UcUWBAf'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.147479", "end": "2021-09-06 11:03:51.362831", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-vzXGpaKj9hB5XLtyL04kPEnXKvJSAFBp5oQdMRHOzXBXtTNBSoWvGbGV4UcUWBAf"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 11:03:51.215352", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144904 s, 796 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144904 s, 796 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:51.432 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.113) 0:04:04.587 ****** 2021-09-06T11:03:51.504 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-09-06T11:03:51.506 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T11:03:51.528 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.095) 0:04:04.683 ****** 2021-09-06T11:03:51.567 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-06T11:03:51.569 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930261512192, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114627, 'block_used': 12980481, 'inode_total': 61054976, 'inode_available': 60986042, 'inode_used': 68934, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114627, "block_size": 4096, "block_total": 240095108, "block_used": 12980481, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986042, "inode_total": 61054976, "inode_used": 68934, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930261512192, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.602 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.073) 0:04:04.757 ****** 2021-09-06T11:03:51.640 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-09-06T11:03:51.641 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.667 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.065) 0:04:04.823 ****** 2021-09-06T11:03:51.685 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-09-06T11:03:51.687 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.052040", "end": "2021-09-06 11:03:51.647442", "rc": 0, "start": "2021-09-06 11:03:51.595402", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:03:51.709 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.041) 0:04:04.864 ****** 2021-09-06T11:03:51.719 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:51.721 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.748 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.039) 0:04:04.903 ****** 2021-09-06T11:03:51.777 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.799 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.051) 0:04:04.955 ****** 2021-09-06T11:03:51.828 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-09-06T11:03:51.830 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.851 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.051) 0:04:05.006 ****** 2021-09-06T11:03:51.890 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-09-06T11:03:51.891 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.912 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.061) 0:04:05.068 ****** 2021-09-06T11:03:51.940 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-09-06T11:03:51.942 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.043759", "end": "2021-09-06 11:03:51.919001", "rc": 0, "start": "2021-09-06 11:03:51.875242", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:51.963 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.051) 0:04:05.119 ****** 2021-09-06T11:03:51.986 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:51 +0000 (0:00:00.021) 0:04:05.140 ****** 2021-09-06T11:03:52.024 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-06T11:03:52.049 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.064) 0:04:05.204 ****** 2021-09-06T11:03:52.087 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-09-06T11:03:52.089 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.112 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.062) 0:04:05.267 ****** 2021-09-06T11:03:52.139 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-09-06T11:03:52.141 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.165 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.052) 0:04:05.320 ****** 2021-09-06T11:03:52.202 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-06T11:03:52.204 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.227 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.062) 0:04:05.383 ****** 2021-09-06T11:03:52.255 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-09-06T11:03:52.257 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.277 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.050) 0:04:05.433 ****** 2021-09-06T11:03:52.473 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-09-06T11:03:52.474 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T11:03:52.495 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.217) 0:04:05.651 ****** 2021-09-06T11:03:52.512 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-09-06T11:03:52.514 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-06T11:03:52.537 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.042) 0:04:05.693 ****** 2021-09-06T11:03:52.545 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-06T11:03:52.547 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930261520384, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114629, 'block_used': 12980479, 'inode_total': 61054976, 'inode_available': 60986042, 'inode_used': 68934, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114629, "block_size": 4096, "block_total": 240095108, "block_used": 12980479, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986042, "inode_total": 61054976, "inode_used": 68934, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930261520384, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.571 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.033) 0:04:05.726 ****** 2021-09-06T11:03:52.611 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.634 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.063) 0:04:05.789 ****** 2021-09-06T11:03:52.636 INFO:teuthology.task.ansible.out: 2021-09-06T11:03:52.673 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:52.675 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.696 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.061) 0:04:05.851 ****** 2021-09-06T11:03:52.723 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:52.725 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.746 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.050) 0:04:05.902 ****** 2021-09-06T11:03:52.774 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-09-06T11:03:52.776 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.797 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.050) 0:04:05.952 ****** 2021-09-06T11:03:52.824 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-09-06T11:03:52.826 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.847 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.049) 0:04:06.002 ****** 2021-09-06T11:03:52.875 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:52.898 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:52 +0000 (0:00:00.051) 0:04:06.053 ****** 2021-09-06T11:03:53.000 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-06T11:03:53.002 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T11:03:53.338 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T11:03:53.340 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-06T11:03:53.366 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:53 +0000 (0:00:00.467) 0:04:06.521 ****** 2021-09-06T11:03:53.744 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-06T11:03:53.746 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T11:03:53.818 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T11:03:54.081 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T11:03:54.135 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T11:03:54.438 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T11:03:54.462 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T11:03:54.487 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:54 +0000 (0:00:01.121) 0:04:07.642 ****** 2021-09-06T11:03:54.682 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-06T11:03:54.684 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-06T11:03:54.707 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:54 +0000 (0:00:00.219) 0:04:07.862 ****** 2021-09-06T11:03:54.766 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-06T11:03:54.768 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T11:03:54.924 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T11:03:54.926 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T11:03:55.096 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T11:03:55.124 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:55 +0000 (0:00:00.417) 0:04:08.280 ****** 2021-09-06T11:03:55.132 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T11:03:55.134 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T11:03:55.273 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T11:03:55.336 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-06T11:03:55.338 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-06T11:03:55.362 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:55 +0000 (0:00:00.237) 0:04:08.517 ****** 2021-09-06T11:03:55.452 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T11:03:55.454 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T11:03:55.464 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:55.490 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:55 +0000 (0:00:00.128) 0:04:08.645 ****** 2021-09-06T11:03:55.518 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-09-06T11:03:55.519 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:55.540 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:55 +0000 (0:00:00.049) 0:04:08.695 ****** 2021-09-06T11:03:55.567 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-09-06T11:03:55.569 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:55.590 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:55 +0000 (0:00:00.050) 0:04:08.746 ****** 2021-09-06T11:03:55.598 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T11:03:55.600 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T11:03:55.746 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T11:03:55.789 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-09-06T11:03:55.791 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T11:03:55.811 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:55 +0000 (0:00:00.220) 0:04:08.966 ****** 2021-09-06T11:03:55.849 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-09-06T11:03:55.851 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-09-06T11:03:55.878 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:55 +0000 (0:00:00.066) 0:04:09.033 ****** 2021-09-06T11:03:55.917 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T11:03:55.919 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T11:03:56.097 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T11:03:56.108 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:56.133 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.255) 0:04:09.289 ****** 2021-09-06T11:03:56.161 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:56.184 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.050) 0:04:09.339 ****** 2021-09-06T11:03:56.190 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-09-06T11:03:56.192 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true} 2021-09-06T11:03:56.217 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.032) 0:04:09.372 ****** 2021-09-06T11:03:56.223 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-09-06T11:03:56.225 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:56.248 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.031) 0:04:09.403 ****** 2021-09-06T11:03:56.445 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-09-06T11:03:56.447 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T11:03:56.467 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.219) 0:04:09.623 ****** 2021-09-06T11:03:56.505 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-09-06T11:03:56.507 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-09-06T11:03:56.530 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.062) 0:04:09.685 ****** 2021-09-06T11:03:56.736 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-06T11:03:56.759 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.229) 0:04:09.915 ****** 2021-09-06T11:03:56.788 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-09-06T11:03:56.789 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:56.809 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.050) 0:04:09.965 ****** 2021-09-06T11:03:56.826 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-09-06T11:03:56.828 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true} 2021-09-06T11:03:56.849 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.039) 0:04:10.005 ****** 2021-09-06T11:03:56.856 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-09-06T11:03:56.858 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:56.882 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.032) 0:04:10.037 ****** 2021-09-06T11:03:56.909 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:56.932 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.050) 0:04:10.087 ****** 2021-09-06T11:03:56.960 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-06T11:03:56.961 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:56.982 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:56 +0000 (0:00:00.050) 0:04:10.138 ****** 2021-09-06T11:03:57.010 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-09-06T11:03:57.012 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.032 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.049) 0:04:10.188 ****** 2021-09-06T11:03:57.059 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-09-06T11:03:57.061 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.082 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.049) 0:04:10.237 ****** 2021-09-06T11:03:57.110 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-09-06T11:03:57.112 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.138 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.056) 0:04:10.293 ****** 2021-09-06T11:03:57.166 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.168 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-06T11:03:57.189 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.050) 0:04:10.344 ****** 2021-09-06T11:03:57.209 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.019) 0:04:10.364 ****** 2021-09-06T11:03:57.216 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-09-06T11:03:57.218 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.240 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.031) 0:04:10.396 ****** 2021-09-06T11:03:57.248 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-09-06T11:03:57.250 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.271 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.030) 0:04:10.427 ****** 2021-09-06T11:03:57.278 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-09-06T11:03:57.280 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.302 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.030) 0:04:10.457 ****** 2021-09-06T11:03:57.309 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-09-06T11:03:57.310 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.332 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.030) 0:04:10.487 ****** 2021-09-06T11:03:57.338 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-09-06T11:03:57.340 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.363 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.030) 0:04:10.518 ****** 2021-09-06T11:03:57.372 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-06T11:03:57.373 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.395 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.032) 0:04:10.551 ****** 2021-09-06T11:03:57.402 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-09-06T11:03:57.404 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.427 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.031) 0:04:10.582 ****** 2021-09-06T11:03:57.435 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-06T11:03:57.437 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.459 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.032) 0:04:10.614 ****** 2021-09-06T11:03:57.466 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-09-06T11:03:57.468 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.491 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.032) 0:04:10.647 ****** 2021-09-06T11:03:57.499 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-09-06T11:03:57.501 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.523 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.031) 0:04:10.678 ****** 2021-09-06T11:03:57.530 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-09-06T11:03:57.532 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.555 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.032) 0:04:10.710 ****** 2021-09-06T11:03:57.562 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-09-06T11:03:57.564 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.586 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.031) 0:04:10.741 ****** 2021-09-06T11:03:57.593 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-09-06T11:03:57.595 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.624 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.037) 0:04:10.779 ****** 2021-09-06T11:03:57.631 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-09-06T11:03:57.633 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.657 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.032) 0:04:10.812 ****** 2021-09-06T11:03:57.684 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-09-06T11:03:57.686 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.716 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.059) 0:04:10.871 ****** 2021-09-06T11:03:57.747 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.770 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.054) 0:04:10.926 ****** 2021-09-06T11:03:57.798 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-09-06T11:03:57.799 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.821 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.050) 0:04:10.976 ****** 2021-09-06T11:03:57.848 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-09-06T11:03:57.850 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.871 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.050) 0:04:11.026 ****** 2021-09-06T11:03:57.898 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-09-06T11:03:57.900 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.925 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.054) 0:04:11.081 ****** 2021-09-06T11:03:57.953 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-09-06T11:03:57.954 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:57.975 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:57 +0000 (0:00:00.049) 0:04:11.130 ****** 2021-09-06T11:03:58.002 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-09-06T11:03:58.004 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:58.025 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.050) 0:04:11.181 ****** 2021-09-06T11:03:58.064 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-09-06T11:03:58.066 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:58.089 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.063) 0:04:11.244 ****** 2021-09-06T11:03:58.096 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-09-06T11:03:58.097 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-09-06T11:03:58.122 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.033) 0:04:11.277 ****** 2021-09-06T11:03:58.436 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-09-06T11:03:58.438 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "867", "ExecMainStartTimestamp": "Mon 2021-09-06 10:59:13 UTC", "ExecMainStartTimestampMonotonic": "12695213", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSI 2021-09-06T11:03:58.440 INFO:teuthology.task.ansible.out:ZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "867", "MemoryAccounting": "yes", "MemoryCurrent": "1830912", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "forking", "UID": "[not set]", "UMask": "0022", "WatchdogTimestamp": "Mon 2021-09-06 1 2021-09-06T11:03:58.442 INFO:teuthology.task.ansible.out:0:59:13 UTC", "WatchdogTimestampMonotonic": "12695247", "WatchdogUSec": "0"}} 2021-09-06T11:03:58.460 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.337) 0:04:11.615 ****** 2021-09-06T11:03:58.487 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:58.489 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-06T11:03:58.510 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.049) 0:04:11.665 ****** 2021-09-06T11:03:58.549 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:58.551 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-06T11:03:58.573 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.063) 0:04:11.729 ****** 2021-09-06T11:03:58.582 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-09-06T11:03:58.583 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-09-06T11:03:58.607 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.034) 0:04:11.763 ****** 2021-09-06T11:03:58.615 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-09-06T11:03:58.617 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:58.640 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.032) 0:04:11.795 ****** 2021-09-06T11:03:58.667 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:58.690 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.050) 0:04:11.845 ****** 2021-09-06T11:03:58.728 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-09-06T11:03:58.730 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:58.752 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.061) 0:04:11.907 ****** 2021-09-06T11:03:58.906 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-09-06T11:03:58.907 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "861", "ExecMainStartTimestamp": "Mon 2021-09-06 10:59:18 UTC", "ExecMainStartTimestampMonotonic": "13051051", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSI 2021-09-06T11:03:58.910 INFO:teuthology.task.ansible.out:ZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "861", "MemoryAccounting": "yes", "MemoryCurrent": "4018176", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "forking", "UID": "[not set]", "UMask": "0022", "WatchdogTimestamp": "Mon 2021-09-06 1 2021-09-06T11:03:58.912 INFO:teuthology.task.ansible.out:0:59:18 UTC", "WatchdogTimestampMonotonic": "13051071", "WatchdogUSec": "0"}} 2021-09-06T11:03:58.936 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.184) 0:04:12.091 ****** 2021-09-06T11:03:58.947 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-09-06T11:03:58.949 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-09-06T11:03:58.973 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:58 +0000 (0:00:00.036) 0:04:12.128 ****** 2021-09-06T11:03:58.981 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:58.983 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-06T11:03:59.007 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:59 +0000 (0:00:00.034) 0:04:12.163 ****** 2021-09-06T11:03:59.046 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:03:59.048 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-06T11:03:59.076 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:59 +0000 (0:00:00.068) 0:04:12.231 ****** 2021-09-06T11:03:59.103 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:59.126 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:59 +0000 (0:00:00.050) 0:04:12.281 ****** 2021-09-06T11:03:59.153 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:59.176 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:59 +0000 (0:00:00.049) 0:04:12.332 ****** 2021-09-06T11:03:59.205 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-09-06T11:03:59.207 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:03:59.227 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:59 +0000 (0:00:00.051) 0:04:12.383 ****** 2021-09-06T11:03:59.422 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-09-06T11:03:59.423 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-09-06T11:03:59.446 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:59 +0000 (0:00:00.218) 0:04:12.601 ****** 2021-09-06T11:03:59.454 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-09-06T11:03:59.480 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:59 +0000 (0:00:00.033) 0:04:12.635 ****** 2021-09-06T11:03:59.920 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-09-06T11:03:59.943 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:03:59 +0000 (0:00:00.463) 0:04:13.099 ****** 2021-09-06T11:04:00.756 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-09-06T11:04:00.758 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-09-06T11:04:00.780 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:00 +0000 (0:00:00.836) 0:04:13.935 ****** 2021-09-06T11:04:00.808 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-09-06T11:04:00.810 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:00.840 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:00 +0000 (0:00:00.060) 0:04:13.996 ****** 2021-09-06T11:04:00.870 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-09-06T11:04:00.872 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-09-06T11:04:00.897 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:00 +0000 (0:00:00.056) 0:04:14.052 ****** 2021-09-06T11:04:00.990 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-09-06T11:04:00.992 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-09-06T11:04:01.016 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.118) 0:04:14.171 ****** 2021-09-06T11:04:01.042 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-09-06T11:04:01.044 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.065 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.049) 0:04:14.220 ****** 2021-09-06T11:04:01.092 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-09-06T11:04:01.094 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} TASK [testnode : Set interface] ************************************************ ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-09-06T11:04:01.117 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.052) 0:04:14.272 ****** 2021-09-06T11:04:01.138 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.020) 0:04:14.293 ****** 2021-09-06T11:04:01.146 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-09-06T11:04:01.147 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.172 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.034) 0:04:14.328 ****** 2021-09-06T11:04:01.202 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-09-06T11:04:01.202 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-06T11:04:01.224 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.052) 0:04:14.380 ****** 2021-09-06T11:04:01.252 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-09-06T11:04:01.254 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.276 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.051) 0:04:14.431 ****** 2021-09-06T11:04:01.325 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-09-06T11:04:01.327 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-09-06T11:04:01.354 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.077) 0:04:14.509 ****** 2021-09-06T11:04:01.383 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-09-06T11:04:01.385 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.413 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.059) 0:04:14.569 ****** 2021-09-06T11:04:01.452 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-09-06T11:04:01.454 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-06T11:04:01.476 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.062) 0:04:14.631 ****** 2021-09-06T11:04:01.483 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-09-06T11:04:01.485 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-06T11:04:01.509 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.032) 0:04:14.664 ****** 2021-09-06T11:04:01.516 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-09-06T11:04:01.518 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.540 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.031) 0:04:14.695 ****** 2021-09-06T11:04:01.696 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-09-06T11:04:01.698 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-06T11:04:01.721 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.180) 0:04:14.876 ****** 2021-09-06T11:04:01.745 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-09-06T11:04:01.747 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-06T11:04:01.768 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.047) 0:04:14.923 ****** 2021-09-06T11:04:01.778 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.800 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.032) 0:04:14.955 ****** 2021-09-06T11:04:01.837 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-09-06T11:04:01.839 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.849 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.851 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.860 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.885 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.084) 0:04:15.040 ****** 2021-09-06T11:04:01.924 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-09-06T11:04:01.925 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.936 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.937 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.947 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:01.972 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.086) 0:04:15.127 ****** 2021-09-06T11:04:01.980 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-09-06T11:04:01.982 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-06T11:04:02.006 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:01 +0000 (0:00:00.034) 0:04:15.161 ****** 2021-09-06T11:04:02.013 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:04:02.014 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.040 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.033) 0:04:15.195 ****** 2021-09-06T11:04:02.047 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-09-06T11:04:02.049 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.072 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.032) 0:04:15.228 ****** 2021-09-06T11:04:02.079 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-09-06T11:04:02.080 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.103 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.030) 0:04:15.259 ****** 2021-09-06T11:04:02.121 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-09-06T11:04:02.124 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.150 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.046) 0:04:15.305 ****** 2021-09-06T11:04:02.158 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-09-06T11:04:02.159 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.185 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.034) 0:04:15.340 ****** 2021-09-06T11:04:02.193 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-09-06T11:04:02.195 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.206 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.207 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.231 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.045) 0:04:15.386 ****** 2021-09-06T11:04:02.238 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-09-06T11:04:02.240 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.264 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.033) 0:04:15.419 ****** 2021-09-06T11:04:02.272 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T11:04:02.274 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.299 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.035) 0:04:15.454 ****** 2021-09-06T11:04:02.308 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-09-06T11:04:02.310 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.336 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.037) 0:04:15.492 ****** 2021-09-06T11:04:02.344 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-09-06T11:04:02.346 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.369 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.032) 0:04:15.525 ****** 2021-09-06T11:04:02.377 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-06T11:04:02.379 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.403 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.033) 0:04:15.558 ****** 2021-09-06T11:04:02.411 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.443 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.040) 0:04:15.598 ****** 2021-09-06T11:04:02.454 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-09-06T11:04:02.456 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.481 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.037) 0:04:15.636 ****** 2021-09-06T11:04:02.488 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-09-06T11:04:02.489 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.513 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.032) 0:04:15.668 ****** 2021-09-06T11:04:02.519 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-06T11:04:02.521 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.543 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.029) 0:04:15.698 ****** 2021-09-06T11:04:02.551 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-09-06T11:04:02.552 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.579 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.036) 0:04:15.734 ****** 2021-09-06T11:04:02.608 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-06T11:04:02.610 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.633 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.053) 0:04:15.788 ****** 2021-09-06T11:04:02.670 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.696 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.063) 0:04:15.851 ****** 2021-09-06T11:04:02.724 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-06T11:04:02.726 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:02.746 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:02 +0000 (0:00:00.050) 0:04:15.902 ****** 2021-09-06T11:04:02.763 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-09-06T11:04:02.765 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004969", "end": "2021-09-06 11:04:02.723345", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-06 11:04:02.718376", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:04:02.931 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003031", "end": "2021-09-06 11:04:02.898219", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-06 11:04:02.895188", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:04:02.963 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004946", "end": "2021-09-06 11:04:02.926962", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-06 11:04:02.922016", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:04:03.101 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002728", "end": "2021-09-06 11:04:03.068325", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-06 11:04:03.065597", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-06T11:04:03.133 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002987", "end": "2021-09-06 11:04:03.100747", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-06 11:04:03.097760", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:04:03.300 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002881", "end": "2021-09-06 11:04:03.269493", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-06 11:04:03.266612", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-06T11:04:03.466 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003130", "end": "2021-09-06 11:04:03.442608", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-06 11:04:03.439478", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:04:03.490 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:03 +0000 (0:00:00.743) 0:04:16.646 ****** 2021-09-06T11:04:03.518 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-06T11:04:03.539 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:03 +0000 (0:00:00.049) 0:04:16.695 ****** 2021-09-06T11:04:03.577 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:04.276 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi094.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:01.004129", "end": "2021-09-06 11:04:04.240665", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-06 11:04:03.236536", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:04:04.298 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:04 +0000 (0:00:00.759) 0:04:17.454 ****** 2021-09-06T11:04:04.326 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-06T11:04:04.348 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:04 +0000 (0:00:00.049) 0:04:17.503 ****** 2021-09-06T11:04:04.375 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-09-06T11:04:04.377 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:04.444 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-09-06T11:04:04.461 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:04 +0000 (0:00:00.112) 0:04:17.616 ****** 2021-09-06T11:04:04.481 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:04 +0000 (0:00:00.020) 0:04:17.637 ****** 2021-09-06T11:04:05.126 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi139.front.sepia.ceph.com] 2021-09-06T11:04:05.175 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:05 +0000 (0:00:00.693) 0:04:18.330 ****** 2021-09-06T11:04:05.202 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T11:04:05.204 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:05.225 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:05 +0000 (0:00:00.049) 0:04:18.380 ****** 2021-09-06T11:04:05.262 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T11:04:05.263 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-09-06T11:04:05.284 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:05 +0000 (0:00:00.059) 0:04:18.440 ****** 2021-09-06T11:04:05.322 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-09-06T11:04:05.324 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-09-06T11:04:05.344 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:05 +0000 (0:00:00.060) 0:04:18.500 ****** 2021-09-06T11:04:06.112 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi094.front.sepia.ceph.com] 2021-09-06T11:04:06.161 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:06 +0000 (0:00:00.816) 0:04:19.317 ****** 2021-09-06T11:04:06.189 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T11:04:06.191 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:06.211 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:06 +0000 (0:00:00.050) 0:04:19.367 ****** 2021-09-06T11:04:06.249 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T11:04:06.251 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-09-06T11:04:06.274 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:06 +0000 (0:00:00.060) 0:04:19.427 ****** 2021-09-06T11:04:06.321 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-09-06T11:04:06.323 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-09-06T11:04:06.344 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:06 +0000 (0:00:00.071) 0:04:19.499 ****** 2021-09-06T11:04:11.100 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-09-06T11:04:11.102 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64", "Installed: podman-docker-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.noarch", "Installed: podman-catatonit-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64", "Removed: podman-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-catatonit-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-docker-3.0.1-7.module_el8.4.0+830+8027e1c4.noarch"]} 2021-09-06T11:04:11.124 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:11 +0000 (0:00:04.780) 0:04:24.279 ****** 2021-09-06T11:04:11.152 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T11:04:11.154 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-09-06T11:04:11.174 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:11 +0000 (0:00:00.050) 0:04:24.330 ****** 2021-09-06T11:04:11.203 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T11:04:11.205 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:11.228 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:11 +0000 (0:00:00.053) 0:04:24.383 ****** 2021-09-06T11:04:11.465 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-09-06T11:04:11.466 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T11:04:11.654 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T11:04:11.680 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:11 +0000 (0:00:00.451) 0:04:24.835 ****** 2021-09-06T11:04:11.919 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-09-06T11:04:11.921 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64", "Installed: podman-docker-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.noarch", "Installed: podman-catatonit-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64", "Removed: podman-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-catatonit-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-docker-3.0.1-7.module_el8.4.0+830+8027e1c4.noarch"]} 2021-09-06T11:04:11.943 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:11 +0000 (0:00:00.262) 0:04:25.098 ****** 2021-09-06T11:04:11.980 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T11:04:11.982 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-09-06T11:04:12.003 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:11 +0000 (0:00:00.060) 0:04:25.159 ****** 2021-09-06T11:04:12.033 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T11:04:12.035 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:12.056 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:12 +0000 (0:00:00.052) 0:04:25.211 ****** 2021-09-06T11:04:12.157 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-09-06T11:04:12.159 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-06T11:04:12.283 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-09-06T11:04:12.285 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T11:04:12.471 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T11:04:12.495 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:12 +0000 (0:00:00.439) 0:04:25.651 ****** 2021-09-06T11:04:12.607 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-09-06T11:04:12.609 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-06T11:04:12.638 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:12 +0000 (0:00:00.142) 0:04:25.793 ****** 2021-09-06T11:04:12.982 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-06T11:04:13.399 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-06T11:04:13.429 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:13 +0000 (0:00:00.791) 0:04:26.584 ****** 2021-09-06T11:04:14.216 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-09-06T11:04:14.218 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:04:14.246 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:14 +0000 (0:00:00.816) 0:04:27.401 ****** 2021-09-06T11:04:14.973 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T11:04:14.996 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:14 +0000 (0:00:00.750) 0:04:28.151 ****** 2021-09-06T11:04:16.659 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-09-06T11:04:16.661 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-qun2igk5-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-qun2igk5-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2021-09-06T11:04:16.663 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-06T11:04:16.686 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:16 +0000 (0:00:01.689) 0:04:29.841 ****** 2021-09-06T11:04:16.723 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-09-06T11:04:16.725 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:16.748 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:16 +0000 (0:00:00.062) 0:04:29.903 ****** 2021-09-06T11:04:16.775 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-09-06T11:04:16.777 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:16.800 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:16 +0000 (0:00:00.051) 0:04:29.955 ****** 2021-09-06T11:04:17.144 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-09-06T11:04:17.146 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.146795", "end": "2021-09-06 11:04:17.116631", "rc": 0, "start": "2021-09-06 11:04:16.969836", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:04:17.168 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:17 +0000 (0:00:00.367) 0:04:30.323 ****** 2021-09-06T11:04:17.197 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:17.386 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-09-06T11:04:17.387 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-v5hns_kq-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-v5hns_kq-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2021-09-06T11:04:17.389 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-06T11:04:17.407 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:17 +0000 (0:00:00.239) 0:04:30.562 ****** 2021-09-06T11:04:17.445 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-09-06T11:04:17.446 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:17.468 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:17 +0000 (0:00:00.061) 0:04:30.623 ****** 2021-09-06T11:04:17.506 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-09-06T11:04:17.508 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:17.528 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:17 +0000 (0:00:00.060) 0:04:30.684 ****** 2021-09-06T11:04:17.883 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-09-06T11:04:17.885 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.152677", "end": "2021-09-06 11:04:17.854892", "rc": 0, "start": "2021-09-06 11:04:17.702215", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T11:04:17.905 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:17 +0000 (0:00:00.376) 0:04:31.060 ****** 2021-09-06T11:04:17.943 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-09-06T11:04:17.945 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T11:04:18.013 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-09-06T11:04:18.014 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-09-06T11:04:18.016 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-09-06T11:04:18.017 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-06T11:04:18.029 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-09-06T11:04:18.031 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:18 +0000 (0:00:00.124) 0:04:31.185 ****** 2021-09-06T11:04:18.690 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] 2021-09-06T11:04:19.718 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] 2021-09-06T11:04:19.736 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** 2021-09-06T11:04:19.739 INFO:teuthology.task.ansible.out:Monday 06 September 2021 11:04:19 +0000 (0:00:01.707) 0:04:32.892 ****** 2021-09-06T11:04:19.945 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-06T11:04:19.967 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-06T11:04:19.969 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* 2021-09-06T11:04:19.971 INFO:teuthology.task.ansible.out:smithi094.front.sepia.ceph.com : ok=132 changed=40 unreachable=0 failed=0 skipped=188 rescued=0 ignored=3 smithi139.front.sepia.ceph.com : ok=132 changed=40 unreachable=0 failed=0 skipped=188 rescued=0 ignored=3 Monday 06 September 2021 11:04:19 +0000 (0:00:00.232) 0:04:33.125 ****** =============================================================================== common : nrpe - Ensure SELinux policy is up to date -------------------- 37.23s Create all admin users with sudo access. ------------------------------- 35.95s users : Update authorized_keys using the keys repo --------------------- 33.29s common : nrpe - Remove SELinux policy package -------------------------- 14.81s common : nrpe - Load SELinux policy package ---------------------------- 14.73s Remove revoked users ---------------------------------------------------- 9.56s testnode : Zap all non-root disks --------------------------------------- 7.17s testnode : Install packages --------------------------------------------- 7.10s container-host : Install container packages ----------------------------- 4.78s testnode : Ensure ceph packages are not present. ------------------------ 4.60s common : Install Kerberos Packages (RedHat) ----------------------------- 4.01s Gathering Facts --------------------------------------------------------- 2.57s testnode : Rebuild rpmdb ------------------------------------------------ 2.53s testnode : Ensure ceph dependency packages are not present. ------------- 2.51s users : Update authorized_keys for each user with literal keys ---------- 1.87s testnode : Configure local mirrorlists ---------------------------------- 1.80s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.78s Gathering Facts --------------------------------------------------------- 1.71s container-host : Install registries-conf-ctl ---------------------------- 1.69s common : Install nrpe package and dependencies (R 2021-09-06T11:04:19.973 INFO:teuthology.task.ansible.out:HEL/CentOS) ------------ 1.62s 2021-09-06T11:04:20.347 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi139.front.sepia.ceph.com')] 2021-09-06T11:04:20.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-06T11:04:20.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T11:04:20.426 DEBUG:teuthology.orchestra.run.smithi094:> true 2021-09-06T11:04:20.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-06T11:04:20.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T11:04:20.578 DEBUG:teuthology.orchestra.run.smithi139:> true 2021-09-06T11:04:20.658 INFO:teuthology.run_tasks:Running task clock... 2021-09-06T11:04:20.670 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-09-06T11:04:20.670 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-06T11:04:20.672 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-06T11:04:20.705 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-09-06T11:04:20.719 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-09-06T11:04:20.755 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2021-09-06T11:04:20.765 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2021-09-06T11:04:20.779 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-09-06T11:04:20.792 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-09-06T11:04:20.853 INFO:teuthology.orchestra.run.smithi094.stderr:bash: ntpq: command not found 2021-09-06T11:04:20.855 INFO:teuthology.orchestra.run.smithi094.stdout:210 Number of sources = 4 2021-09-06T11:04:20.856 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-09-06T11:04:20.856 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2021-09-06T11:04:20.856 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T11:04:20.856 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T11:04:20.857 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T11:04:20.857 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T11:04:20.857 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-06T11:04:20.858 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-06T11:04:20.890 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-09-06T11:04:20.905 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-09-06T11:04:20.937 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2021-09-06T11:04:20.947 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2021-09-06T11:04:20.961 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-09-06T11:04:20.974 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-09-06T11:04:21.039 INFO:teuthology.orchestra.run.smithi139.stderr:bash: ntpq: command not found 2021-09-06T11:04:21.041 INFO:teuthology.orchestra.run.smithi139.stdout:210 Number of sources = 4 2021-09-06T11:04:21.042 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-09-06T11:04:21.042 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2021-09-06T11:04:21.042 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T11:04:21.042 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T11:04:21.043 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T11:04:21.043 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T11:04:21.043 INFO:teuthology.run_tasks:Running task install... 2021-09-06T11:04:21.053 DEBUG:teuthology.task.install:project ceph 2021-09-06T11:04:21.053 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e'}} 2021-09-06T11:04:21.054 DEBUG:teuthology.task.install:config {'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e'} 2021-09-06T11:04:21.054 INFO:teuthology.task.install:Using flavor: basic 2021-09-06T11:04:21.061 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-09-06T11:04:21.061 INFO:teuthology.task.install:extra packages: [] 2021-09-06T11:04:21.062 DEBUG:teuthology.orchestra.run.smithi094:> sudo lsb_release -is 2021-09-06T11:04:21.095 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS 2021-09-06T11:04:21.096 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-06T11:04:21.097 DEBUG:teuthology.orchestra.run.smithi139:> sudo lsb_release -is 2021-09-06T11:04:21.097 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'tag': None, 'wait_for_package': False} 2021-09-06T11:04:21.098 DEBUG:teuthology.orchestra.run.smithi094:> sudo lsb_release -is 2021-09-06T11:04:21.130 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS 2021-09-06T11:04:21.132 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-06T11:04:21.132 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'tag': None, 'wait_for_package': False} 2021-09-06T11:04:21.133 DEBUG:teuthology.orchestra.run.smithi139:> sudo lsb_release -is 2021-09-06T11:04:21.167 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS 2021-09-06T11:04:21.169 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-06T11:04:21.169 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2227f2382f1df5117a62687acd2d99ea5eeb871e 2021-09-06T11:04:21.210 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS 2021-09-06T11:04:21.211 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-06T11:04:21.211 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2227f2382f1df5117a62687acd2d99ea5eeb871e 2021-09-06T11:04:21.327 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-vstart/2227f2382f1df5117a62687acd2d99ea5eeb871e/centos/8/flavors/default/ 2021-09-06T11:04:21.327 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7472.g2227f238 2021-09-06T11:04:21.364 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-vstart/2227f2382f1df5117a62687acd2d99ea5eeb871e/centos/8/flavors/default/ 2021-09-06T11:04:21.364 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7472.g2227f238 2021-09-06T11:04:21.459 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-vstart/2227f2382f1df5117a62687acd2d99ea5eeb871e/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-vstart/2227f2382f1df5117a62687acd2d99ea5eeb871e/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-vstart/2227f2382f1df5117a62687acd2d99ea5eeb871e/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-09-06T11:04:21.459 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:04:21.460 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-09-06T11:04:21.483 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-09-06T11:04:21.483 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2227f2382f1df5117a62687acd2d99ea5eeb871e/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-09-06T11:04:21.492 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-vstart/2227f2382f1df5117a62687acd2d99ea5eeb871e/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-vstart/2227f2382f1df5117a62687acd2d99ea5eeb871e/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-vstart/2227f2382f1df5117a62687acd2d99ea5eeb871e/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-09-06T11:04:21.493 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:04:21.493 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-09-06T11:04:21.518 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-09-06T11:04:21.519 DEBUG:teuthology.orchestra.run.smithi139:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2227f2382f1df5117a62687acd2d99ea5eeb871e/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-09-06T11:04:21.548 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-09-06T11:04:21.584 DEBUG:teuthology.orchestra.run.smithi139:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-09-06T11:04:21.621 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-09-06T11:04:21.657 DEBUG:teuthology.orchestra.run.smithi139:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-09-06T11:04:21.684 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2021-09-06T11:04:21.685 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2021-09-06T11:04:21.720 INFO:teuthology.orchestra.run.smithi139.stdout:check_obsoletes = 1 2021-09-06T11:04:21.721 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum clean all 2021-09-06T11:04:21.915 INFO:teuthology.orchestra.run.smithi094.stdout:84 files removed 2021-09-06T11:04:21.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2021-09-06T11:04:21.951 INFO:teuthology.orchestra.run.smithi139.stdout:84 files removed 2021-09-06T11:04:21.971 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw 2021-09-06T11:04:22.495 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 328 kB/s | 76 kB 00:00 2021-09-06T11:04:22.523 INFO:teuthology.orchestra.run.smithi139.stdout:ceph packages for x86_64 335 kB/s | 76 kB 00:00 2021-09-06T11:04:22.731 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 36 kB/s | 7.4 kB 00:00 2021-09-06T11:04:22.758 INFO:teuthology.orchestra.run.smithi139.stdout:ceph noarch packages 36 kB/s | 7.4 kB 00:00 2021-09-06T11:04:22.960 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 7.0 kB/s | 1.4 kB 00:00 2021-09-06T11:04:22.976 INFO:teuthology.orchestra.run.smithi139.stdout:ceph source packages 7.4 kB/s | 1.4 kB 00:00 2021-09-06T11:04:23.199 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-8 - AppStream 41 MB/s | 8.8 MB 00:00 2021-09-06T11:04:23.285 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - AppStream 31 MB/s | 8.8 MB 00:00 2021-09-06T11:04:24.868 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Base 27 MB/s | 5.6 MB 00:00 2021-09-06T11:04:24.876 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-8 - Base 19 MB/s | 5.6 MB 00:00 2021-09-06T11:04:25.720 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-8 - Extras 161 kB/s | 10 kB 00:00 2021-09-06T11:04:25.730 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Extras 135 kB/s | 10 kB 00:00 2021-09-06T11:04:27.847 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Linux 8 - AppStream 4.3 MB/s | 8.8 MB 00:02 2021-09-06T11:04:27.878 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Linux 8 - AppStream 4.2 MB/s | 8.8 MB 00:02 2021-09-06T11:11:33.799 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Linux 8 - BaseOS 13 kB/s | 5.6 MB 07:04 2021-09-06T11:11:34.952 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Linux 8 - Extras 28 kB/s | 10 kB 00:00 2021-09-06T11:11:35.095 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-8 - PowerTools 27 MB/s | 2.3 MB 00:00 2021-09-06T11:11:37.774 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-09-06T11:11:38.968 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 10 MB 00:01 2021-09-06T11:11:41.146 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-09-06T11:11:43.356 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:11:43.358 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:11:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2021-09-06T11:11:43.359 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:11:43.359 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:11:43.359 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:17.0.0-7472.g2227f238.el8 ceph 8.4 M 2021-09-06T11:11:43.360 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2021-09-06T11:11:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:17.0.0-7472.g2227f238.el8 ceph 3.4 M 2021-09-06T11:11:43.360 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:17.0.0-7472.g2227f238.el8 ceph 3.9 M 2021-09-06T11:11:43.360 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2021-09-06T11:11:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:17.0.0-7472.g2227f238.el8 ceph 5.2 M 2021-09-06T11:11:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:17.0.0-7472.g2227f238.el8 ceph 20 M 2021-09-06T11:11:43.361 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:17.0.0-7472.g2227f238.el8 ceph 24 k 2021-09-06T11:11:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-09-06T11:11:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-09-06T11:11:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:17.0.0-7472.g2227f238.el8 ceph 783 k 2021-09-06T11:11:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-09-06T11:11:43.362 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-09-06T11:11:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:17.0.0-7472.g2227f238.el8 ceph 482 k 2021-09-06T11:11:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-09-06T11:11:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:17.0.0-7472.g2227f238.el8 ceph 3.7 M 2021-09-06T11:11:43.363 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-09-06T11:11:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-09-06T11:11:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse 2021-09-06T11:11:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: x86_64 2:17.0.0-7472.g2227f238.el8 ceph 45 k 2021-09-06T11:11:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:17.0.0-7472.g2227f238.el8 ceph 78 k 2021-09-06T11:11:43.364 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:17.0.0-7472.g2227f238.el8 ceph 178 k 2021-09-06T11:11:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-09-06T11:11:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:17.0.0-7472.g2227f238.el8 ceph 365 k 2021-09-06T11:11:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:17.0.0-7472.g2227f238.el8 ceph 346 k 2021-09-06T11:11:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:17.0.0-7472.g2227f238.el8 ceph 110 k 2021-09-06T11:11:43.366 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:11:43.366 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:11:43.366 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:11:43.366 INFO:teuthology.orchestra.run.smithi094.stdout:Install 21 Packages 2021-09-06T11:11:43.366 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2021-09-06T11:11:43.367 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:11:43.367 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 48 M 2021-09-06T11:11:43.368 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:11:44.662 INFO:teuthology.orchestra.run.smithi094.stdout:(1/23): ceph-common-17.0.0-7472.g2227f238.el8.x 15 MB/s | 20 MB 00:01 2021-09-06T11:11:44.693 INFO:teuthology.orchestra.run.smithi094.stdout:(2/23): ceph-selinux-17.0.0-7472.g2227f238.el8. 768 kB/s | 24 kB 00:00 2021-09-06T11:11:44.744 INFO:teuthology.orchestra.run.smithi094.stdout:(3/23): libcephfs2-17.0.0-7472.g2227f238.el8.x8 15 MB/s | 783 kB 00:00 2021-09-06T11:11:44.782 INFO:teuthology.orchestra.run.smithi094.stdout:(4/23): libradosstriper1-17.0.0-7472.g2227f238. 12 MB/s | 482 kB 00:00 2021-09-06T11:11:44.822 INFO:teuthology.orchestra.run.smithi094.stdout:(5/23): ceph-radosgw-17.0.0-7472.g2227f238.el8. 5.7 MB/s | 8.4 MB 00:01 2021-09-06T11:11:44.854 INFO:teuthology.orchestra.run.smithi094.stdout:(6/23): ceph-base-17.0.0-7472.g2227f238.el8.x86 3.5 MB/s | 5.2 MB 00:01 2021-09-06T11:11:44.873 INFO:teuthology.orchestra.run.smithi094.stdout:(7/23): python3-ceph-argparse-17.0.0-7472.g2227 880 kB/s | 45 kB 00:00 2021-09-06T11:11:44.907 INFO:teuthology.orchestra.run.smithi094.stdout:(8/23): python3-ceph-common-17.0.0-7472.g2227f2 1.4 MB/s | 78 kB 00:00 2021-09-06T11:11:44.930 INFO:teuthology.orchestra.run.smithi094.stdout:(9/23): python3-cephfs-17.0.0-7472.g2227f238.el 3.1 MB/s | 178 kB 00:00 2021-09-06T11:11:44.951 INFO:teuthology.orchestra.run.smithi094.stdout:(10/23): librgw2-17.0.0-7472.g2227f238.el8.x86_ 22 MB/s | 3.7 MB 00:00 2021-09-06T11:11:44.964 INFO:teuthology.orchestra.run.smithi094.stdout:(11/23): python3-rbd-17.0.0-7472.g2227f238.el8. 10 MB/s | 346 kB 00:00 2021-09-06T11:11:44.973 INFO:teuthology.orchestra.run.smithi094.stdout:(12/23): python3-rgw-17.0.0-7472.g2227f238.el8. 4.9 MB/s | 110 kB 00:00 2021-09-06T11:11:44.979 INFO:teuthology.orchestra.run.smithi094.stdout:(13/23): python3-rados-17.0.0-7472.g2227f238.el 4.9 MB/s | 365 kB 00:00 2021-09-06T11:11:44.998 INFO:teuthology.orchestra.run.smithi094.stdout:(14/23): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2021-09-06T11:11:45.000 INFO:teuthology.orchestra.run.smithi094.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 29 MB/s | 47 kB 00:00 2021-09-06T11:11:45.007 INFO:teuthology.orchestra.run.smithi094.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.4 MB/s | 259 kB 00:00 2021-09-06T11:11:45.008 INFO:teuthology.orchestra.run.smithi094.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 1.5 MB/s | 44 kB 00:00 2021-09-06T11:11:45.120 INFO:teuthology.orchestra.run.smithi094.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 526 kB/s | 59 kB 00:00 2021-09-06T11:11:45.125 INFO:teuthology.orchestra.run.smithi094.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 1.5 MB/s | 181 kB 00:00 2021-09-06T11:11:45.147 INFO:teuthology.orchestra.run.smithi094.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 2.0 MB/s | 306 kB 00:00 2021-09-06T11:11:45.176 INFO:teuthology.orchestra.run.smithi094.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2021-09-06T11:11:45.325 INFO:teuthology.orchestra.run.smithi094.stdout:(22/23): librbd1-17.0.0-7472.g2227f238.el8.x86_ 22 MB/s | 3.9 MB 00:00 2021-09-06T11:11:45.568 INFO:teuthology.orchestra.run.smithi094.stdout:(23/23): librados2-17.0.0-7472.g2227f238.el8.x8 7.7 MB/s | 3.4 MB 00:00 2021-09-06T11:11:45.572 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:11:45.572 INFO:teuthology.orchestra.run.smithi094.stdout:Total 22 MB/s | 48 MB 00:02 2021-09-06T11:11:45.576 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:11:45.620 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:11:45.620 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:11:45.999 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:11:46.002 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:11:46.969 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:11:47.156 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2021-09-06T11:11:47.298 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-09-06T11:11:47.710 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-09-06T11:11:47.840 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:17.0.0-7472.g2227f238.el8.x86_64 3/25 2021-09-06T11:11:47.923 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:17.0.0-7472.g2227f238.el8.x86_64 3/25 2021-09-06T11:11:48.122 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:17.0.0-7472.g2227f238.el8.x86_64 4/25 2021-09-06T11:11:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:17.0.0-7472.g2227f238.el8.x86_64 5/25 2021-09-06T11:11:48.654 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:17.0.0-7472.g2227f238.el8.x86_64 5/25 2021-09-06T11:11:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:17.0.0-7472.g2227f238.el8.x86_64 6/25 2021-09-06T11:11:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:17.0.0-7472.g2227f238.el8.x86_64 6/25 2021-09-06T11:11:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-09-06T11:11:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-06T11:11:49.574 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-06T11:11:49.706 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:17.0.0-7472.g2227f238.el8.x86_64 9/25 2021-09-06T11:11:49.786 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:17.0.0-7472.g2227f238.el8.x86_64 9/25 2021-09-06T11:11:49.905 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-09-06T11:11:50.058 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:17.0.0-7472.g2227f238.el8. 11/25 2021-09-06T11:11:50.210 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:17.0.0-7472.g2227f238.el8.x86_64 12/25 2021-09-06T11:11:50.440 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:17.0.0-7472.g2227f238.el8.x86_64 13/25 2021-09-06T11:11:50.624 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:17.0.0-7472.g2227f238.el8.x86_64 14/25 2021-09-06T11:11:50.765 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:17.0.0-7472.g2227f238.el8.x86_6 15/25 2021-09-06T11:11:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7472.g2227f238.el8.x86_6 15/25 2021-09-06T11:11:51.024 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2021-09-06T11:11:51.240 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2021-09-06T11:11:51.375 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2021-09-06T11:11:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:17.0.0-7472.g2227f238.el8.x8 19/25 2021-09-06T11:11:55.513 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 20/25 2021-09-06T11:11:55.689 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 20/25 2021-09-06T11:11:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 20/25 2021-09-06T11:11:56.474 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:17.0.0-7472.g2227f238.el8.x86_64 21/25 2021-09-06T11:11:56.570 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:17.0.0-7472.g2227f238.el8.x86_64 21/25 2021-09-06T11:11:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:17.0.0-7472.g2227f238.el8.x86_64 22/25 2021-09-06T11:12:12.214 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7472.g2227f238.el8.x86_64 22/25 2021-09-06T11:12:12.215 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2021-09-06T11:12:12.215 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2021-09-06T11:12:12.215 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /dev 2021-09-06T11:12:12.215 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /run 2021-09-06T11:12:12.216 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2021-09-06T11:12:12.216 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2021-09-06T11:12:12.216 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2021-09-06T11:12:12.216 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2021-09-06T11:12:12.216 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /dev 2021-09-06T11:12:12.217 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:13.505 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:17.0.0-7472.g2227f238.el8.x86_64 23/25 2021-09-06T11:12:13.585 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7472.g2227f238.el8.x86_64 23/25 2021-09-06T11:12:13.586 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-06T11:12:13.773 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-06T11:12:13.773 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T11:12:14.438 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T11:12:14.439 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:17.0.0-7472.g2227f238.el8.x86_64 1/25 2021-09-06T11:12:14.439 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 2/25 2021-09-06T11:12:14.439 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:17.0.0-7472.g2227f238.el8.x86_64 3/25 2021-09-06T11:12:14.439 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:17.0.0-7472.g2227f238.el8.x86_64 4/25 2021-09-06T11:12:14.440 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:17.0.0-7472.g2227f238.el8.x86_64 5/25 2021-09-06T11:12:14.440 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:17.0.0-7472.g2227f238.el8.x86_6 6/25 2021-09-06T11:12:14.440 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:17.0.0-7472.g2227f238.el8.x86_64 7/25 2021-09-06T11:12:14.440 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7472.g2227f238.el8. 8/25 2021-09-06T11:12:14.440 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:17.0.0-7472.g2227f238.el8.x8 9/25 2021-09-06T11:12:14.441 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:17.0.0-7472.g2227f238.el8.x86_64 10/25 2021-09-06T11:12:14.441 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:17.0.0-7472.g2227f238.el8.x86_64 11/25 2021-09-06T11:12:14.441 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:17.0.0-7472.g2227f238.el8.x86_64 12/25 2021-09-06T11:12:14.441 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:17.0.0-7472.g2227f238.el8.x86_64 13/25 2021-09-06T11:12:14.442 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2021-09-06T11:12:14.442 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2021-09-06T11:12:14.442 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-09-06T11:12:14.442 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2021-09-06T11:12:14.442 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-09-06T11:12:14.443 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2021-09-06T11:12:14.443 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2021-09-06T11:12:14.443 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2021-09-06T11:12:14.443 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:17.0.0-7472.g2227f238.el8.x86_64 22/25 2021-09-06T11:12:14.444 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-09-06T11:12:14.444 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:17.0.0-7472.g2227f238.el8.x86_64 24/25 2021-09-06T11:12:15.115 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T11:12:15.115 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:15.116 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2021-09-06T11:12:15.116 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.116 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.116 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:15.117 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:12:15.117 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.117 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.117 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.118 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.118 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-09-06T11:12:15.118 INFO:teuthology.orchestra.run.smithi094.stdout: leveldb-1.22-1.el8.x86_64 2021-09-06T11:12:15.118 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.119 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.2-3.el8.x86_64 2021-09-06T11:12:15.119 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-09-06T11:12:15.119 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.119 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-09-06T11:12:15.120 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.120 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.3.1-3.el8.x86_64 2021-09-06T11:12:15.120 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-09-06T11:12:15.120 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.120 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.121 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.121 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-09-06T11:12:15.121 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.121 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.122 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:15.122 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:15.122 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:12:15.331 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2021-09-06T11:12:15.789 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:34 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:12:16.730 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:12:16.731 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:12:16.732 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2021-09-06T11:12:16.732 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:12:16.732 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:12:16.732 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:17.0.0-7472.g2227f238.el8 ceph 43 M 2021-09-06T11:12:16.733 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2021-09-06T11:12:16.733 INFO:teuthology.orchestra.run.smithi094.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-09-06T11:12:16.733 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:17.0.0-7472.g2227f238.el8 ceph 170 k 2021-09-06T11:12:16.733 INFO:teuthology.orchestra.run.smithi094.stdout: libxslt x86_64 1.1.32-6.el8 CentOS-Base 250 k 2021-09-06T11:12:16.734 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-09-06T11:12:16.734 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-09-06T11:12:16.734 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-09-06T11:12:16.734 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:16.735 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:12:16.735 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:12:16.735 INFO:teuthology.orchestra.run.smithi094.stdout:Install 7 Packages 2021-09-06T11:12:16.735 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:16.736 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 44 M 2021-09-06T11:12:16.736 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 207 M 2021-09-06T11:12:16.736 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:12:16.771 INFO:teuthology.orchestra.run.smithi094.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 4.2 MB/s | 161 kB 00:00 2021-09-06T11:12:16.775 INFO:teuthology.orchestra.run.smithi094.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 57 MB/s | 187 kB 00:00 2021-09-06T11:12:16.779 INFO:teuthology.orchestra.run.smithi094.stdout:(3/7): socat-1.7.3.3-2.el8.x86_64.rpm 65 MB/s | 302 kB 00:00 2021-09-06T11:12:16.784 INFO:teuthology.orchestra.run.smithi094.stdout:(4/7): libxslt-1.1.32-6.el8.x86_64.rpm 50 MB/s | 250 kB 00:00 2021-09-06T11:12:16.886 INFO:teuthology.orchestra.run.smithi094.stdout:(5/7): libcephsqlite-17.0.0-7472.g2227f238.el8. 1.1 MB/s | 170 kB 00:00 2021-09-06T11:12:16.961 INFO:teuthology.orchestra.run.smithi094.stdout:(6/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 390 kB/s | 69 kB 00:00 2021-09-06T11:12:25.406 INFO:teuthology.orchestra.run.smithi094.stdout:(7/7): ceph-test-17.0.0-7472.g2227f238.el8.x86_ 5.0 MB/s | 43 MB 00:08 2021-09-06T11:12:25.410 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:12:25.411 INFO:teuthology.orchestra.run.smithi094.stdout:Total 5.1 MB/s | 44 MB 00:08 2021-09-06T11:12:25.415 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:12:25.438 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:12:25.439 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:12:25.682 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:12:25.685 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:12:26.198 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:12:26.354 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libxslt-1.1.32-6.el8.x86_64 1/7 2021-09-06T11:12:26.522 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-09-06T11:12:26.688 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-09-06T11:12:26.844 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-06T11:12:26.905 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-06T11:12:27.092 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-09-06T11:12:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:17.0.0-7472.g2227f238.el8.x86_64 6/7 2021-09-06T11:12:31.473 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7472.g2227f238.el8.x86_64 6/7 2021-09-06T11:12:31.766 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:17.0.0-7472.g2227f238.el8.x86_64 7/7 2021-09-06T11:12:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:17.0.0-7472.g2227f238.el8.x86_64 7/7 2021-09-06T11:12:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:17.0.0-7472.g2227f238.el8.x86_64 1/7 2021-09-06T11:12:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:17.0.0-7472.g2227f238.el8.x86_64 2/7 2021-09-06T11:12:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2021-09-06T11:12:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-06T11:12:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/7 2021-09-06T11:12:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libxslt-1.1.32-6.el8.x86_64 6/7 2021-09-06T11:12:32.404 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2021-09-06T11:12:32.405 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:32.406 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:12:32.406 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:17.0.0-7472.g2227f238.el8.x86_64 jq-1.5-12.el8.x86_64 2021-09-06T11:12:32.406 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:17.0.0-7472.g2227f238.el8.x86_64 libxslt-1.1.32-6.el8.x86_64 2021-09-06T11:12:32.407 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-09-06T11:12:32.407 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-09-06T11:12:32.407 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:32.407 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:12:32.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2021-09-06T11:12:33.002 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:52 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:12:33.992 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:12:33.996 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2021-09-06T11:12:33.996 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2021-09-06T11:12:33.997 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2021-09-06T11:12:33.997 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:12:33.997 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:17.0.0-7472.g2227f238.el8 ceph 6.4 k 2021-09-06T11:12:33.997 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2021-09-06T11:12:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:17.0.0-7472.g2227f238.el8 ceph 2.3 M 2021-09-06T11:12:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:17.0.0-7472.g2227f238.el8 ceph 1.5 M 2021-09-06T11:12:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 214 k 2021-09-06T11:12:33.998 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:17.0.0-7472.g2227f238.el8 ceph 3.9 M 2021-09-06T11:12:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:17.0.0-7472.g2227f238.el8 ceph 17 M 2021-09-06T11:12:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-09-06T11:12:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-09-06T11:12:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-09-06T11:12:33.999 INFO:teuthology.orchestra.run.smithi094.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-09-06T11:12:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-09-06T11:12:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-09-06T11:12:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 3.2.1-4.el8 CentOS-Base 559 k 2021-09-06T11:12:34.000 INFO:teuthology.orchestra.run.smithi094.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-09-06T11:12:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-09-06T11:12:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-09-06T11:12:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-09-06T11:12:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: python3-lxml x86_64 4.2.3-2.el8 CentOS-AppStream 1.5 M 2021-09-06T11:12:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-09-06T11:12:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-09-06T11:12:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-09-06T11:12:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-09-06T11:12:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-09-06T11:12:34.002 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-09-06T11:12:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-09-06T11:12:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-09-06T11:12:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-09-06T11:12:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-09-06T11:12:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-09-06T11:12:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-09-06T11:12:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-09-06T11:12:34.004 INFO:teuthology.orchestra.run.smithi094.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-09-06T11:12:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-09-06T11:12:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-09-06T11:12:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-09-06T11:12:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-09-06T11:12:34.005 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-09-06T11:12:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-09-06T11:12:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-09-06T11:12:34.006 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2021-09-06T11:12:34.006 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-09-06T11:12:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-09-06T11:12:34.007 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:34.007 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:12:34.007 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2021-09-06T11:12:34.007 INFO:teuthology.orchestra.run.smithi094.stdout:Install 41 Packages 2021-09-06T11:12:34.008 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:34.008 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 31 M 2021-09-06T11:12:34.009 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 121 M 2021-09-06T11:12:34.009 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:12:34.105 INFO:teuthology.orchestra.run.smithi094.stdout:(1/41): ceph-17.0.0-7472.g2227f238.el8.x86_64.r 62 kB/s | 6.4 kB 00:00 2021-09-06T11:12:34.287 INFO:teuthology.orchestra.run.smithi094.stdout:(2/41): ceph-mgr-17.0.0-7472.g2227f238.el8.x86_ 5.3 MB/s | 1.5 MB 00:00 2021-09-06T11:12:34.403 INFO:teuthology.orchestra.run.smithi094.stdout:(3/41): ceph-mds-17.0.0-7472.g2227f238.el8.x86_ 5.6 MB/s | 2.3 MB 00:00 2021-09-06T11:12:34.434 INFO:teuthology.orchestra.run.smithi094.stdout:(4/41): ceph-mgr-modules-core-17.0.0-7472.g2227 6.8 MB/s | 214 kB 00:00 2021-09-06T11:12:34.465 INFO:teuthology.orchestra.run.smithi094.stdout:(5/41): python3-html5lib-0.999999999-6.el8.noar 6.7 MB/s | 214 kB 00:00 2021-09-06T11:12:34.484 INFO:teuthology.orchestra.run.smithi094.stdout:(6/41): python3-lxml-4.2.3-2.el8.x86_64.rpm 81 MB/s | 1.5 MB 00:00 2021-09-06T11:12:34.486 INFO:teuthology.orchestra.run.smithi094.stdout:(7/41): python3-mako-1.0.6-13.el8.noarch.rpm 59 MB/s | 157 kB 00:00 2021-09-06T11:12:34.488 INFO:teuthology.orchestra.run.smithi094.stdout:(8/41): python3-markupsafe-0.23-19.el8.x86_64.r 35 MB/s | 39 kB 00:00 2021-09-06T11:12:34.490 INFO:teuthology.orchestra.run.smithi094.stdout:(9/41): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 56 MB/s | 103 kB 00:00 2021-09-06T11:12:34.491 INFO:teuthology.orchestra.run.smithi094.stdout:(10/41): python3-pytz-2017.2-9.el8.noarch.rpm 40 MB/s | 54 kB 00:00 2021-09-06T11:12:34.492 INFO:teuthology.orchestra.run.smithi094.stdout:(11/41): python3-webencodings-0.5.1-6.el8.noarc 25 MB/s | 27 kB 00:00 2021-09-06T11:12:34.499 INFO:teuthology.orchestra.run.smithi094.stdout:(12/41): python3-werkzeug-0.12.2-4.el8.noarch.r 73 MB/s | 457 kB 00:00 2021-09-06T11:12:34.502 INFO:teuthology.orchestra.run.smithi094.stdout:(13/41): python3-cffi-1.11.5-5.el8.x86_64.rpm 68 MB/s | 237 kB 00:00 2021-09-06T11:12:34.505 INFO:teuthology.orchestra.run.smithi094.stdout:(14/41): python3-chardet-3.0.4-7.el8.noarch.rpm 66 MB/s | 195 kB 00:00 2021-09-06T11:12:34.513 INFO:teuthology.orchestra.run.smithi094.stdout:(15/41): python3-cryptography-3.2.1-4.el8.x86_6 76 MB/s | 559 kB 00:00 2021-09-06T11:12:34.515 INFO:teuthology.orchestra.run.smithi094.stdout:(16/41): python3-idna-2.5-5.el8.noarch.rpm 52 MB/s | 97 kB 00:00 2021-09-06T11:12:34.517 INFO:teuthology.orchestra.run.smithi094.stdout:(17/41): python3-pycparser-2.14-14.el8.noarch.r 58 MB/s | 109 kB 00:00 2021-09-06T11:12:34.518 INFO:teuthology.orchestra.run.smithi094.stdout:(18/41): python3-pysocks-1.6.8-3.el8.noarch.rpm 33 MB/s | 34 kB 00:00 2021-09-06T11:12:34.521 INFO:teuthology.orchestra.run.smithi094.stdout:(19/41): python3-requests-2.20.0-2.1.el8_1.noar 50 MB/s | 123 kB 00:00 2021-09-06T11:12:34.523 INFO:teuthology.orchestra.run.smithi094.stdout:(20/41): python3-urllib3-1.24.2-5.el8.noarch.rp 64 MB/s | 177 kB 00:00 2021-09-06T11:12:34.558 INFO:teuthology.orchestra.run.smithi094.stdout:(21/41): ceph-mon-17.0.0-7472.g2227f238.el8.x86 8.5 MB/s | 3.9 MB 00:00 2021-09-06T11:12:34.750 INFO:teuthology.orchestra.run.smithi094.stdout:(22/41): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 193 kB/s | 44 kB 00:00 2021-09-06T11:12:34.833 INFO:teuthology.orchestra.run.smithi094.stdout:(23/41): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.0 MB/s | 173 kB 00:00 2021-09-06T11:12:34.869 INFO:teuthology.orchestra.run.smithi094.stdout:(24/41): python3-beautifulsoup4-4.6.3-2.el8.1.n 596 kB/s | 185 kB 00:00 2021-09-06T11:12:34.894 INFO:teuthology.orchestra.run.smithi094.stdout:(25/41): python3-cssselect-0.9.2-10.el8.noarch. 1.6 MB/s | 40 kB 00:00 2021-09-06T11:12:34.916 INFO:teuthology.orchestra.run.smithi094.stdout:(26/41): python3-cherrypy-18.4.0-1.el8.noarch.r 4.5 MB/s | 384 kB 00:00 2021-09-06T11:12:34.918 INFO:teuthology.orchestra.run.smithi094.stdout:(27/41): python3-jaraco-6.2-6.el8.noarch.rpm 456 kB/s | 11 kB 00:00 2021-09-06T11:12:34.944 INFO:teuthology.orchestra.run.smithi094.stdout:(28/41): python3-logutils-0.3.5-11.el8.noarch.r 1.9 MB/s | 49 kB 00:00 2021-09-06T11:12:34.944 INFO:teuthology.orchestra.run.smithi094.stdout:(29/41): python3-jaraco-functools-2.0-4.el8.noa 634 kB/s | 18 kB 00:00 2021-09-06T11:12:34.969 INFO:teuthology.orchestra.run.smithi094.stdout:(30/41): python3-more-itertools-7.2.0-3.el8.noa 2.2 MB/s | 59 kB 00:00 2021-09-06T11:12:34.994 INFO:teuthology.orchestra.run.smithi094.stdout:(31/41): python3-portend-2.6-1.el8.noarch.rpm 637 kB/s | 16 kB 00:00 2021-09-06T11:12:35.001 INFO:teuthology.orchestra.run.smithi094.stdout:(32/41): python3-pecan-1.3.2-9.el8.noarch.rpm 4.9 MB/s | 283 kB 00:00 2021-09-06T11:12:35.019 INFO:teuthology.orchestra.run.smithi094.stdout:(33/41): python3-simplegeneric-0.8.1-17.el8.noa 757 kB/s | 19 kB 00:00 2021-09-06T11:12:35.146 INFO:teuthology.orchestra.run.smithi094.stdout:(34/41): ceph-osd-17.0.0-7472.g2227f238.el8.x86 20 MB/s | 17 MB 00:00 2021-09-06T11:12:35.146 INFO:teuthology.orchestra.run.smithi094.stdout:(35/41): python3-singledispatch-3.4.0.3-18.el8. 165 kB/s | 24 kB 00:00 2021-09-06T11:12:35.147 INFO:teuthology.orchestra.run.smithi094.stdout:(36/41): python3-tempora-1.14.1-5.el8.noarch.rp 229 kB/s | 29 kB 00:00 2021-09-06T11:12:35.197 INFO:teuthology.orchestra.run.smithi094.stdout:(37/41): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.9 MB/s | 251 kB 00:00 2021-09-06T11:12:35.203 INFO:teuthology.orchestra.run.smithi094.stdout:(38/41): python3-waitress-1.2.1-2.el8.1.noarch. 4.1 MB/s | 241 kB 00:00 2021-09-06T11:12:35.222 INFO:teuthology.orchestra.run.smithi094.stdout:(39/41): python3-webtest-2.0.33-1.el8.noarch.rp 3.3 MB/s | 85 kB 00:00 2021-09-06T11:12:35.231 INFO:teuthology.orchestra.run.smithi094.stdout:(40/41): python3-zc-lockfile-2.0-2.el8.noarch.r 823 kB/s | 23 kB 00:00 2021-09-06T11:12:35.267 INFO:teuthology.orchestra.run.smithi094.stdout:(41/41): python3-trustme-0.6.0-4.el8.noarch.rpm 227 kB/s | 27 kB 00:00 2021-09-06T11:12:35.271 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:12:35.271 INFO:teuthology.orchestra.run.smithi094.stdout:Total 24 MB/s | 31 MB 00:01 2021-09-06T11:12:35.284 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:12:35.325 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:12:35.326 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:12:35.648 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:12:35.651 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:12:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:12:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/41 2021-09-06T11:12:37.554 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/41 2021-09-06T11:12:37.717 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-idna-2.5-5.el8.noarch 3/41 2021-09-06T11:12:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/41 2021-09-06T11:12:38.133 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/41 2021-09-06T11:12:38.288 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/41 2021-09-06T11:12:38.436 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/41 2021-09-06T11:12:38.637 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/41 2021-09-06T11:12:38.756 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/41 2021-09-06T11:12:38.912 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/41 2021-09-06T11:12:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/41 2021-09-06T11:12:39.273 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/41 2021-09-06T11:12:39.533 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 13/41 2021-09-06T11:12:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/41 2021-09-06T11:12:40.137 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-09-06T11:12:40.421 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 16/41 2021-09-06T11:12:40.575 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 17/41 2021-09-06T11:12:40.776 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 18/41 2021-09-06T11:12:40.977 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 19/41 2021-09-06T11:12:41.193 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 20/41 2021-09-06T11:12:41.668 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 21/41 2021-09-06T11:12:41.912 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 22/41 2021-09-06T11:12:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/41 2021-09-06T11:12:42.363 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 24/41 2021-09-06T11:12:42.563 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 25/41 2021-09-06T11:12:42.930 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 26/41 2021-09-06T11:12:43.144 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-lxml-4.2.3-2.el8.x86_64 27/41 2021-09-06T11:12:43.365 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 28/41 2021-09-06T11:12:43.512 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 29/41 2021-09-06T11:12:43.672 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 30/41 2021-09-06T11:12:43.920 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-2.6-1.el8.noarch 31/41 2021-09-06T11:12:44.121 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 32/41 2021-09-06T11:12:44.355 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/41 2021-09-06T11:12:44.700 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/41 2021-09-06T11:12:44.998 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 35/41 2021-09-06T11:12:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7472.g2227f238.el8. 36/41 2021-09-06T11:12:45.568 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:17.0.0-7472.g2227f238.el8.x86_64 37/41 2021-09-06T11:12:47.178 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7472.g2227f238.el8.x86_64 37/41 2021-09-06T11:12:47.616 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:17.0.0-7472.g2227f238.el8.x86_64 38/41 2021-09-06T11:12:48.017 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:17.0.0-7472.g2227f238.el8.x86_64 38/41 2021-09-06T11:12:48.266 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:17.0.0-7472.g2227f238.el8.x86_64 39/41 2021-09-06T11:12:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:17.0.0-7472.g2227f238.el8.x86_64 39/41 2021-09-06T11:12:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:17.0.0-7472.g2227f238.el8.x86_64 40/41 2021-09-06T11:12:48.649 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:17.0.0-7472.g2227f238.el8.x86_64 40/41 2021-09-06T11:12:48.773 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:17.0.0-7472.g2227f238.el8.x86_64 41/41 2021-09-06T11:12:49.132 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:17.0.0-7472.g2227f238.el8.x86_64 41/41 2021-09-06T11:12:49.132 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:17.0.0-7472.g2227f238.el8.x86_64 1/41 2021-09-06T11:12:49.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:17.0.0-7472.g2227f238.el8.x86_64 2/41 2021-09-06T11:12:49.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:17.0.0-7472.g2227f238.el8.x86_64 3/41 2021-09-06T11:12:49.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:17.0.0-7472.g2227f238.el8.x86_64 4/41 2021-09-06T11:12:49.133 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:17.0.0-7472.g2227f238.el8.x86_64 5/41 2021-09-06T11:12:49.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7472.g2227f238.el8. 6/41 2021-09-06T11:12:49.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/41 2021-09-06T11:12:49.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-lxml-4.2.3-2.el8.x86_64 8/41 2021-09-06T11:12:49.134 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/41 2021-09-06T11:12:49.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/41 2021-09-06T11:12:49.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 11/41 2021-09-06T11:12:49.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/41 2021-09-06T11:12:49.135 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/41 2021-09-06T11:12:49.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/41 2021-09-06T11:12:49.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-09-06T11:12:49.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 16/41 2021-09-06T11:12:49.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 17/41 2021-09-06T11:12:49.136 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-idna-2.5-5.el8.noarch 18/41 2021-09-06T11:12:49.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 19/41 2021-09-06T11:12:49.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 20/41 2021-09-06T11:12:49.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 21/41 2021-09-06T11:12:49.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 22/41 2021-09-06T11:12:49.137 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 23/41 2021-09-06T11:12:49.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/41 2021-09-06T11:12:49.138 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 25/41 2021-09-06T11:12:49.140 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 26/41 2021-09-06T11:12:49.140 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 27/41 2021-09-06T11:12:49.140 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 28/41 2021-09-06T11:12:49.141 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 29/41 2021-09-06T11:12:49.141 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 30/41 2021-09-06T11:12:49.141 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 31/41 2021-09-06T11:12:49.141 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 32/41 2021-09-06T11:12:49.141 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-2.6-1.el8.noarch 33/41 2021-09-06T11:12:49.142 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 34/41 2021-09-06T11:12:49.142 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 35/41 2021-09-06T11:12:49.142 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 36/41 2021-09-06T11:12:49.142 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 37/41 2021-09-06T11:12:49.143 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 38/41 2021-09-06T11:12:49.143 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 39/41 2021-09-06T11:12:49.143 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 40/41 2021-09-06T11:12:50.034 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 41/41 2021-09-06T11:12:50.034 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:50.035 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:12:50.035 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:50.035 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:50.035 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:50.035 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:12:50.036 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:50.036 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:50.036 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-09-06T11:12:50.036 INFO:teuthology.orchestra.run.smithi094.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-09-06T11:12:50.037 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-09-06T11:12:50.037 INFO:teuthology.orchestra.run.smithi094.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-09-06T11:12:50.037 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-09-06T11:12:50.037 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-09-06T11:12:50.037 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-09-06T11:12:50.038 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-09-06T11:12:50.038 INFO:teuthology.orchestra.run.smithi094.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-09-06T11:12:50.038 INFO:teuthology.orchestra.run.smithi094.stdout: python3-idna-2.5-5.el8.noarch 2021-09-06T11:12:50.038 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-6.2-6.el8.noarch 2021-09-06T11:12:50.039 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-09-06T11:12:50.039 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-09-06T11:12:50.039 INFO:teuthology.orchestra.run.smithi094.stdout: python3-lxml-4.2.3-2.el8.x86_64 2021-09-06T11:12:50.039 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.0.6-13.el8.noarch 2021-09-06T11:12:50.039 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-09-06T11:12:50.040 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-09-06T11:12:50.040 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-09-06T11:12:50.040 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-2.6-1.el8.noarch 2021-09-06T11:12:50.040 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-09-06T11:12:50.041 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.14-14.el8.noarch 2021-09-06T11:12:50.041 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-09-06T11:12:50.041 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2017.2-9.el8.noarch 2021-09-06T11:12:50.041 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-09-06T11:12:50.041 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-09-06T11:12:50.042 INFO:teuthology.orchestra.run.smithi094.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-09-06T11:12:50.042 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-09-06T11:12:50.042 INFO:teuthology.orchestra.run.smithi094.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-09-06T11:12:50.042 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-09-06T11:12:50.043 INFO:teuthology.orchestra.run.smithi094.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-09-06T11:12:50.043 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-09-06T11:12:50.043 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-09-06T11:12:50.043 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-09-06T11:12:50.043 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-09-06T11:12:50.044 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-09-06T11:12:50.044 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:50.044 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:12:50.219 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2021-09-06T11:12:50.668 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:09 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:12:51.361 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:12:51.611 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:12:51.612 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2021-09-06T11:12:51.612 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:12:51.664 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2021-09-06T11:12:52.105 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:11 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:12:53.069 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:12:53.070 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:12:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2021-09-06T11:12:53.070 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:12:53.071 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:12:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 69 k 2021-09-06T11:12:53.071 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:53.071 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:12:53.072 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:12:53.072 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2021-09-06T11:12:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:53.072 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 69 k 2021-09-06T11:12:53.072 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 309 k 2021-09-06T11:12:53.073 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:12:53.203 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-17.0.0-7472.g2227f238.el8.noarch.rpm 524 kB/s | 69 kB 00:00 2021-09-06T11:12:53.204 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:12:53.204 INFO:teuthology.orchestra.run.smithi094.stdout:Total 519 kB/s | 69 kB 00:00 2021-09-06T11:12:53.204 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:12:53.211 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:12:53.211 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:12:53.212 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:12:53.216 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:12:53.259 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:12:55.369 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:17.0.0-7472.g2227f238.el8.noarch 1/1 2021-09-06T11:12:55.613 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:17.0.0-7472.g2227f238.el8.noarch 1/1 2021-09-06T11:12:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:17.0.0-7472.g2227f238.el8.noarch 1/1 2021-09-06T11:12:56.119 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:17.0.0-7472.g2227f238.el8.noarch 1/1 2021-09-06T11:12:56.119 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:56.119 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:12:56.120 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:12:56.120 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:56.120 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:12:56.243 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2021-09-06T11:12:56.701 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:15 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:12:57.659 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:12:57.660 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:12:57.661 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2021-09-06T11:12:57.661 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:12:57.661 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:12:57.661 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7472.g2227f238.el8 ceph 169 k 2021-09-06T11:12:57.662 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:57.662 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:12:57.662 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:12:57.662 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2021-09-06T11:12:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:57.663 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 169 k 2021-09-06T11:12:57.663 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 567 k 2021-09-06T11:12:57.663 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:12:57.863 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-17.0.0-7472.g2227f2 837 kB/s | 169 kB 00:00 2021-09-06T11:12:57.863 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:12:57.864 INFO:teuthology.orchestra.run.smithi094.stdout:Total 830 kB/s | 169 kB 00:00 2021-09-06T11:12:57.865 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:12:57.877 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:12:57.877 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:12:57.926 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:12:57.929 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:12:58.166 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:12:58.348 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7472.g2227f238. 1/1 2021-09-06T11:12:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7472.g2227f238. 1/1 2021-09-06T11:12:58.975 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7472.g2227f238. 1/1 2021-09-06T11:12:58.975 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:58.976 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:12:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:12:58.976 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:12:58.977 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:12:59.113 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2021-09-06T11:12:59.559 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:18 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:00.259 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:13:00.505 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:00.506 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2021-09-06T11:13:00.507 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:00.542 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2021-09-06T11:13:00.984 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:19 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:01.968 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:01.968 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:01.969 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2021-09-06T11:13:01.969 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:01.969 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:13:01.969 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 1.5 M 2021-09-06T11:13:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2021-09-06T11:13:01.970 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 24 k 2021-09-06T11:13:01.970 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 8.9 k 2021-09-06T11:13:01.970 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-09-06T11:13:01.971 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-09-06T11:13:01.971 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-09-06T11:13:01.971 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:01.971 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:13:01.972 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:01.972 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2021-09-06T11:13:01.972 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:01.973 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.8 M 2021-09-06T11:13:01.973 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 7.4 M 2021-09-06T11:13:01.973 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:13:02.062 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7472.g2227 98 kB/s | 8.9 kB 00:00 2021-09-06T11:13:02.094 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2021-09-06T11:13:02.096 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): ceph-grafana-dashboards-17.0.0-7472.g222 191 kB/s | 24 kB 00:00 2021-09-06T11:13:02.320 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 149 kB/s | 34 kB 00:00 2021-09-06T11:13:02.347 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): ceph-mgr-dashboard-17.0.0-7472.g2227f238 4.1 MB/s | 1.5 MB 00:00 2021-09-06T11:13:02.376 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 701 kB/s | 196 kB 00:00 2021-09-06T11:13:02.379 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:13:02.380 INFO:teuthology.orchestra.run.smithi094.stdout:Total 4.5 MB/s | 1.8 MB 00:00 2021-09-06T11:13:02.381 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:13:02.388 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:13:02.389 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:13:02.455 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:13:02.458 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:13:02.728 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:13:02.922 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-09-06T11:13:03.117 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-09-06T11:13:03.467 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-09-06T11:13:05.015 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7472.g2227f238.el8.n 4/6 2021-09-06T11:13:05.361 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7472.g2227f238.el8. 5/6 2021-09-06T11:13:05.573 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7472.g2227f238.el8.noarc 6/6 2021-09-06T11:13:05.828 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7472.g2227f238.el8.noarc 6/6 2021-09-06T11:13:05.828 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7472.g2227f238.el8. 1/6 2021-09-06T11:13:05.828 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7472.g2227f238.el8.noarc 2/6 2021-09-06T11:13:05.829 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7472.g2227f238.el8.n 3/6 2021-09-06T11:13:05.829 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-09-06T11:13:05.829 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-09-06T11:13:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-09-06T11:13:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:06.101 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:13:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:13:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:13:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:13:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-09-06T11:13:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-09-06T11:13:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.4.1-12.el8.noarch 2021-09-06T11:13:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:06.103 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:06.157 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2021-09-06T11:13:06.598 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:25 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:07.536 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:07.537 INFO:teuthology.orchestra.run.smithi094.stdout:===================================================================================================== 2021-09-06T11:13:07.538 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2021-09-06T11:13:07.538 INFO:teuthology.orchestra.run.smithi094.stdout:===================================================================================================== 2021-09-06T11:13:07.538 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:13:07.539 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 7.4 M 2021-09-06T11:13:07.539 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2021-09-06T11:13:07.539 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 8.4.1-1.el8 CentOS-Base 639 k 2021-09-06T11:13:07.539 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 8.4.1-1.el8 CentOS-Base 169 k 2021-09-06T11:13:07.539 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.12-1.el8 CentOS-AppStream 4.6 M 2021-09-06T11:13:07.540 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-threads x86_64 0.3.12-1.el8 CentOS-AppStream 4.7 M 2021-09-06T11:13:07.540 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-09-06T11:13:07.540 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-09-06T11:13:07.540 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.0.0-20.module_el8.4.0+790+083e3d81 CentOS-AppStream 15 M 2021-09-06T11:13:07.541 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:07.541 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:13:07.541 INFO:teuthology.orchestra.run.smithi094.stdout:===================================================================================================== 2021-09-06T11:13:07.541 INFO:teuthology.orchestra.run.smithi094.stdout:Install 8 Packages 2021-09-06T11:13:07.542 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:07.543 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 36 M 2021-09-06T11:13:07.543 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 197 M 2021-09-06T11:13:07.543 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:13:07.764 INFO:teuthology.orchestra.run.smithi094.stdout:(1/8): openblas-0.3.12-1.el8.x86_64.rpm 20 MB/s | 4.6 MB 00:00 2021-09-06T11:13:07.872 INFO:teuthology.orchestra.run.smithi094.stdout:(2/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2021-09-06T11:13:07.895 INFO:teuthology.orchestra.run.smithi094.stdout:(3/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 29 MB/s | 3.7 MB 00:00 2021-09-06T11:13:07.896 INFO:teuthology.orchestra.run.smithi094.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 9.1 MB/s | 225 kB 00:00 2021-09-06T11:13:07.933 INFO:teuthology.orchestra.run.smithi094.stdout:(5/8): libgfortran-8.4.1-1.el8.x86_64.rpm 17 MB/s | 639 kB 00:00 2021-09-06T11:13:07.943 INFO:teuthology.orchestra.run.smithi094.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 15 MB/s | 169 kB 00:00 2021-09-06T11:13:08.186 INFO:teuthology.orchestra.run.smithi094.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.4.0+79 50 MB/s | 15 MB 00:00 2021-09-06T11:13:09.438 INFO:teuthology.orchestra.run.smithi094.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-747 3.9 MB/s | 7.4 MB 00:01 2021-09-06T11:13:09.441 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:13:09.442 INFO:teuthology.orchestra.run.smithi094.stdout:Total 19 MB/s | 36 MB 00:01 2021-09-06T11:13:09.451 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:13:09.472 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:13:09.472 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:13:09.845 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:13:09.848 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:13:10.585 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:13:10.730 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-06T11:13:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-06T11:13:10.996 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-06T11:13:11.507 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-06T11:13:11.672 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.12-1.el8.x86_64 3/8 2021-09-06T11:13:12.216 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 3/8 2021-09-06T11:13:12.330 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-06T11:13:13.095 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-06T11:13:13.479 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-09-06T11:13:13.679 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-06T11:13:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-06T11:13:17.887 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 7/8 2021-09-06T11:13:18.063 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7472.g2227f23 8/8 2021-09-06T11:13:18.369 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7472.g2227f23 8/8 2021-09-06T11:13:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7472.g2227f23 1/8 2021-09-06T11:13:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 2/8 2021-09-06T11:13:18.370 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-06T11:13:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-09-06T11:13:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-09-06T11:13:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 6/8 2021-09-06T11:13:18.371 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 7/8 2021-09-06T11:13:18.665 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 8/8 2021-09-06T11:13:18.665 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:18.666 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:13:18.666 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:13:18.666 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-09-06T11:13:18.666 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-09-06T11:13:18.667 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.12-1.el8.x86_64 2021-09-06T11:13:18.667 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-09-06T11:13:18.667 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-09-06T11:13:18.667 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-09-06T11:13:18.667 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x86_64 2021-09-06T11:13:18.668 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:18.668 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:18.723 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2021-09-06T11:13:19.165 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:38 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:20.108 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:20.109 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2021-09-06T11:13:20.110 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2021-09-06T11:13:20.110 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2021-09-06T11:13:20.110 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:13:20.111 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 44 k 2021-09-06T11:13:20.111 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2021-09-06T11:13:20.111 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-09-06T11:13:20.111 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-09-06T11:13:20.112 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-09-06T11:13:20.112 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-09-06T11:13:20.112 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-09-06T11:13:20.112 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-09-06T11:13:20.112 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-09-06T11:13:20.113 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-09-06T11:13:20.113 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-09-06T11:13:20.113 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-09-06T11:13:20.113 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-09-06T11:13:20.114 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-09-06T11:13:20.114 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:20.114 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:13:20.114 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2021-09-06T11:13:20.115 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2021-09-06T11:13:20.115 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:20.116 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.9 M 2021-09-06T11:13:20.116 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 25 M 2021-09-06T11:13:20.116 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:13:20.144 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 854 kB/s | 27 kB 00:00 2021-09-06T11:13:20.147 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 37 MB/s | 126 kB 00:00 2021-09-06T11:13:20.148 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 569 kB/s | 20 kB 00:00 2021-09-06T11:13:20.150 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 34 MB/s | 110 kB 00:00 2021-09-06T11:13:20.151 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 43 MB/s | 155 kB 00:00 2021-09-06T11:13:20.152 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 33 MB/s | 43 kB 00:00 2021-09-06T11:13:20.245 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): ceph-mgr-rook-17.0.0-7472.g2227f238.el8 334 kB/s | 44 kB 00:00 2021-09-06T11:13:20.268 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 284 kB/s | 33 kB 00:00 2021-09-06T11:13:20.273 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 135 kB/s | 16 kB 00:00 2021-09-06T11:13:20.338 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-rsa-3.4.2-16.el8.noarch.rpm 1.1 MB/s | 73 kB 00:00 2021-09-06T11:13:20.376 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.6 MB/s | 61 kB 00:00 2021-09-06T11:13:20.450 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-google-auth-1.1.1-10.el8.noarc 395 kB/s | 81 kB 00:00 2021-09-06T11:13:20.527 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.5 MB/s | 1.2 MB 00:00 2021-09-06T11:13:20.529 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:13:20.530 INFO:teuthology.orchestra.run.smithi094.stdout:Total 4.6 MB/s | 1.9 MB 00:00 2021-09-06T11:13:20.535 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:13:20.548 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:13:20.548 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:13:20.621 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:13:20.624 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:13:21.105 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:13:21.325 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-09-06T11:13:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-09-06T11:13:21.699 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-09-06T11:13:21.903 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-09-06T11:13:22.128 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-09-06T11:13:22.585 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-09-06T11:13:22.921 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-09-06T11:13:23.129 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-09-06T11:13:24.225 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-09-06T11:13:24.461 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-09-06T11:13:24.627 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-09-06T11:13:24.813 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-09-06T11:13:25.046 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:17.0.0-7472.g2227f238.el8.noarch 13/13 2021-09-06T11:13:25.283 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7472.g2227f238.el8.noarch 13/13 2021-09-06T11:13:25.283 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7472.g2227f238.el8.noarch 1/13 2021-09-06T11:13:25.284 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-09-06T11:13:25.284 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-09-06T11:13:25.284 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-09-06T11:13:25.284 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-09-06T11:13:25.285 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-09-06T11:13:25.285 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-09-06T11:13:25.285 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-09-06T11:13:25.285 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-09-06T11:13:25.286 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-09-06T11:13:25.286 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-09-06T11:13:25.286 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-09-06T11:13:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-09-06T11:13:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:25.624 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:13:25.624 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:13:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-09-06T11:13:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-09-06T11:13:25.625 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-09-06T11:13:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-09-06T11:13:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-09-06T11:13:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-09-06T11:13:25.626 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-09-06T11:13:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-09-06T11:13:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-09-06T11:13:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-09-06T11:13:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-09-06T11:13:25.627 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-09-06T11:13:25.628 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:25.628 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:25.698 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2021-09-06T11:13:26.140 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:45 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:27.113 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:27.114 INFO:teuthology.orchestra.run.smithi094.stdout:=================================================================================================================== 2021-09-06T11:13:27.114 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2021-09-06T11:13:27.114 INFO:teuthology.orchestra.run.smithi094.stdout:=================================================================================================================== 2021-09-06T11:13:27.114 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:13:27.115 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 91 k 2021-09-06T11:13:27.115 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2021-09-06T11:13:27.115 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.7.0-1.el8 copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 476 k 2021-09-06T11:13:27.115 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-09-06T11:13:27.116 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-09-06T11:13:27.116 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:27.116 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:13:27.116 INFO:teuthology.orchestra.run.smithi094.stdout:=================================================================================================================== 2021-09-06T11:13:27.116 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2021-09-06T11:13:27.117 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:27.117 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 5.9 M 2021-09-06T11:13:27.117 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 25 M 2021-09-06T11:13:27.117 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:13:27.163 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 11 MB/s | 538 kB 00:00 2021-09-06T11:13:27.214 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): python3-babel-2.5.1-5.el8.noarch.rpm 49 MB/s | 4.8 MB 00:00 2021-09-06T11:13:27.246 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): ceph-mgr-cephadm-17.0.0-7472.g2227f238.e 694 kB/s | 91 kB 00:00 2021-09-06T11:13:29.012 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): python3-asyncssh-2.7.0-1.el8.noarch.rpm 257 kB/s | 476 kB 00:01 2021-09-06T11:13:29.017 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:13:29.017 INFO:teuthology.orchestra.run.smithi094.stdout:Total 3.1 MB/s | 5.9 MB 00:01 2021-09-06T11:13:29.021 INFO:teuthology.orchestra.run.smithi094.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ceph:python3-asyncssh-5dfbfa110d1e0dad/packages/python3-asyncssh-2.7.0-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 45bf1189: NOKEY 2021-09-06T11:13:29.840 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for python3-asyncssh owned by ceph 1.2 kB/s | 1.0 kB 00:00 2021-09-06T11:13:29.952 INFO:teuthology.orchestra.run.smithi094.stderr:Importing GPG key 0x45BF1189: 2021-09-06T11:13:29.953 INFO:teuthology.orchestra.run.smithi094.stderr: Userid : "ceph_python3-asyncssh (None) " 2021-09-06T11:13:29.953 INFO:teuthology.orchestra.run.smithi094.stderr: Fingerprint: FD53 FC11 9FA9 2C47 E35D 9C35 339B 8950 45BF 1189 2021-09-06T11:13:29.953 INFO:teuthology.orchestra.run.smithi094.stderr: From : https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/pubkey.gpg 2021-09-06T11:13:30.044 INFO:teuthology.orchestra.run.smithi094.stdout:Key imported successfully 2021-09-06T11:13:30.046 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:13:30.055 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:13:30.056 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:13:30.141 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:13:30.144 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:13:30.470 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:13:31.458 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.7.0-1.el8.noarch 1/4 2021-09-06T11:13:31.821 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 2/4 2021-09-06T11:13:32.088 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-06T11:13:32.269 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7472.g2227f238.el8.noarch 4/4 2021-09-06T11:13:32.504 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7472.g2227f238.el8.noarch 4/4 2021-09-06T11:13:32.504 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7472.g2227f238.el8.noarch 1/4 2021-09-06T11:13:32.504 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/4 2021-09-06T11:13:32.504 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-06T11:13:32.798 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.7.0-1.el8.noarch 4/4 2021-09-06T11:13:32.798 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:32.798 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:13:32.799 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:13:32.799 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.7.0-1.el8.noarch 2021-09-06T11:13:32.799 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.5.1-5.el8.noarch 2021-09-06T11:13:32.799 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-09-06T11:13:32.800 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:32.800 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:32.855 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2021-09-06T11:13:33.301 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:52 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:34.237 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:34.238 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:34.238 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2021-09-06T11:13:34.238 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:34.239 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:13:34.239 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:17.0.0-7472.g2227f238.el8 ceph 868 k 2021-09-06T11:13:34.239 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2021-09-06T11:13:34.239 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-09-06T11:13:34.240 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:34.240 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:13:34.240 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:34.240 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2021-09-06T11:13:34.241 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:34.241 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 950 k 2021-09-06T11:13:34.241 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 3.1 M 2021-09-06T11:13:34.241 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:13:34.274 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.3 MB/s | 82 kB 00:00 2021-09-06T11:13:34.875 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-fuse-17.0.0-7472.g2227f238.el8.x86_ 1.3 MB/s | 868 kB 00:00 2021-09-06T11:13:34.876 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:13:34.877 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.5 MB/s | 950 kB 00:00 2021-09-06T11:13:34.878 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:13:34.899 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:13:34.900 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:13:34.978 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:13:34.981 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:13:35.157 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:13:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-09-06T11:13:35.710 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 2/2 2021-09-06T11:13:36.061 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 2/2 2021-09-06T11:13:36.061 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 1/2 2021-09-06T11:13:36.325 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-09-06T11:13:36.325 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:36.325 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:13:36.325 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-09-06T11:13:36.326 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:36.326 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:36.523 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2021-09-06T11:13:36.974 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:55 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:37.926 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:37.927 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:37.927 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2021-09-06T11:13:37.928 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:37.928 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:13:37.928 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:17.0.0-7472.g2227f238.el8 ceph 125 k 2021-09-06T11:13:37.928 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:37.929 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:13:37.929 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:37.929 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2021-09-06T11:13:37.929 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:37.930 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 125 k 2021-09-06T11:13:37.930 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 432 k 2021-09-06T11:13:37.930 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:13:38.079 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-17.0.0-7472.g2227f238.el8.x86_64 824 kB/s | 125 kB 00:00 2021-09-06T11:13:38.080 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:13:38.080 INFO:teuthology.orchestra.run.smithi094.stdout:Total 817 kB/s | 125 kB 00:00 2021-09-06T11:13:38.081 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:13:38.087 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:13:38.087 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:13:38.136 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:13:38.139 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:13:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:13:38.484 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:38.712 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:38.987 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:38.988 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:13:38.988 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:13:38.989 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:38.989 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:39.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2021-09-06T11:13:39.588 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:58 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:40.296 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:13:40.546 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:40.547 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2021-09-06T11:13:40.547 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:40.591 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2021-09-06T11:13:41.035 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:00 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:41.970 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:41.970 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:41.970 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2021-09-06T11:13:41.971 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:41.971 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:13:41.971 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:17.0.0-7472.g2227f238.el8 ceph 24 k 2021-09-06T11:13:41.971 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:41.972 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:13:41.972 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:41.972 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2021-09-06T11:13:41.972 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:41.973 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 24 k 2021-09-06T11:13:41.973 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 102 k 2021-09-06T11:13:41.973 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:13:42.083 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-17.0.0-7472.g2227f238.el8.x86_6 211 kB/s | 24 kB 00:00 2021-09-06T11:13:42.084 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:13:42.085 INFO:teuthology.orchestra.run.smithi094.stdout:Total 209 kB/s | 24 kB 00:00 2021-09-06T11:13:42.085 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:13:42.087 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:13:42.088 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:13:42.089 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:13:42.092 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:13:42.184 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:13:42.341 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:42.551 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:42.803 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:42.804 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:42.804 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:13:42.804 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:13:42.804 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:42.805 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:42.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2021-09-06T11:13:43.309 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:02 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:44.405 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:13:44.657 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:44.658 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2021-09-06T11:13:44.659 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:44.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2021-09-06T11:13:45.269 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:04 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:45.972 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:13:46.218 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:46.219 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2021-09-06T11:13:46.220 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:46.263 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2021-09-06T11:13:46.707 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:05 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:47.395 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:13:47.645 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:47.646 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2021-09-06T11:13:47.647 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:47.688 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2021-09-06T11:13:48.132 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:07 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:48.828 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:13:49.078 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:49.079 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2021-09-06T11:13:49.080 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:49.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2021-09-06T11:13:49.568 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:08 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:50.986 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:13:51.237 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:51.238 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2021-09-06T11:13:51.239 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:51.287 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2021-09-06T11:13:51.729 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:10 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:52.428 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:13:52.678 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:52.679 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2021-09-06T11:13:52.680 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:52.721 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2021-09-06T11:13:53.163 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:12 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:54.143 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:54.144 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:54.144 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2021-09-06T11:13:54.144 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:54.144 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:13:54.145 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:17.0.0-7472.g2227f238.el8 ceph 93 k 2021-09-06T11:13:54.145 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:54.145 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:13:54.145 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:54.146 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2021-09-06T11:13:54.146 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:54.146 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 93 k 2021-09-06T11:13:54.146 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 281 k 2021-09-06T11:13:54.147 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:13:54.329 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-17.0.0-7472.g2227f238.el8.x86_64.rpm 506 kB/s | 93 kB 00:00 2021-09-06T11:13:54.330 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:13:54.331 INFO:teuthology.orchestra.run.smithi094.stdout:Total 502 kB/s | 93 kB 00:00 2021-09-06T11:13:54.331 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:13:54.341 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:13:54.341 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:13:54.390 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:13:54.393 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:13:54.495 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:13:54.698 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:54.941 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:55.219 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:55.220 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:55.220 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:13:55.220 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:13:55.221 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:55.221 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:55.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2021-09-06T11:13:55.792 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:14 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:13:56.740 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:13:56.741 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2021-09-06T11:13:56.742 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:56.742 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:13:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:17.0.0-7472.g2227f238.el8 ceph 3.8 M 2021-09-06T11:13:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:56.743 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:13:56.743 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:13:56.744 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2021-09-06T11:13:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:56.744 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.8 M 2021-09-06T11:13:56.744 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 19 M 2021-09-06T11:13:56.745 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:13:58.185 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-17.0.0-7472.g2227f238.el8.x86_64.rpm 2.6 MB/s | 3.8 MB 00:01 2021-09-06T11:13:58.186 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:13:58.187 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.6 MB/s | 3.8 MB 00:01 2021-09-06T11:13:58.187 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:13:58.202 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:13:58.202 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:13:58.270 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:13:58.273 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:13:58.748 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:13:58.933 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:59.301 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:13:59.591 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:59.591 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:13:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:13:59.592 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:13:59.592 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:13:59.781 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2021-09-06T11:14:00.232 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:19 ago on Mon 06 Sep 2021 11:11:41 AM UTC. 2021-09-06T11:14:01.183 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2021-09-06T11:14:01.184 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:14:01.184 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2021-09-06T11:14:01.185 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:14:01.185 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2021-09-06T11:14:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:17.0.0-7472.g2227f238.el8 ceph 173 k 2021-09-06T11:14:01.185 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:14:01.186 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2021-09-06T11:14:01.186 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2021-09-06T11:14:01.186 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2021-09-06T11:14:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:14:01.186 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 173 k 2021-09-06T11:14:01.187 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 563 k 2021-09-06T11:14:01.187 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2021-09-06T11:14:01.455 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-17.0.0-7472.g2227f238.el8.x86_64.rpm 640 kB/s | 173 kB 00:00 2021-09-06T11:14:01.456 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:14:01.457 INFO:teuthology.orchestra.run.smithi094.stdout:Total 637 kB/s | 173 kB 00:00 2021-09-06T11:14:01.457 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2021-09-06T11:14:01.468 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2021-09-06T11:14:01.469 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2021-09-06T11:14:01.518 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2021-09-06T11:14:01.521 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2021-09-06T11:14:01.670 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2021-09-06T11:14:01.872 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:14:02.122 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:14:02.627 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:14:02.628 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:14:02.628 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2021-09-06T11:14:02.628 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:14:02.628 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:14:02.629 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2021-09-06T11:14:02.786 DEBUG:teuthology.parallel:result is None 2021-09-06T11:20:12.587 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Linux 8 - BaseOS 6.0 kB/s | 5.6 MB 15:43 2021-09-06T11:20:13.914 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Linux 8 - Extras 19 kB/s | 10 kB 00:00 2021-09-06T11:20:14.058 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - PowerTools 26 MB/s | 2.3 MB 00:00 2021-09-06T11:20:16.771 INFO:teuthology.orchestra.run.smithi139.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-09-06T11:20:18.258 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 7.2 MB/s | 10 MB 00:01 2021-09-06T11:20:20.446 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2021-09-06T11:20:22.692 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:20:22.695 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:20:22.695 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-09-06T11:20:22.696 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:20:22.696 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:20:22.696 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw x86_64 2:17.0.0-7472.g2227f238.el8 ceph 8.4 M 2021-09-06T11:20:22.696 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2021-09-06T11:20:22.696 INFO:teuthology.orchestra.run.smithi139.stdout: librados2 x86_64 2:17.0.0-7472.g2227f238.el8 ceph 3.4 M 2021-09-06T11:20:22.697 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1 x86_64 2:17.0.0-7472.g2227f238.el8 ceph 3.9 M 2021-09-06T11:20:22.697 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-09-06T11:20:22.697 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base x86_64 2:17.0.0-7472.g2227f238.el8 ceph 5.2 M 2021-09-06T11:20:22.697 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common x86_64 2:17.0.0-7472.g2227f238.el8 ceph 20 M 2021-09-06T11:20:22.698 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux x86_64 2:17.0.0-7472.g2227f238.el8 ceph 24 k 2021-09-06T11:20:22.698 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-09-06T11:20:22.698 INFO:teuthology.orchestra.run.smithi139.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-09-06T11:20:22.698 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2 x86_64 2:17.0.0-7472.g2227f238.el8 ceph 783 k 2021-09-06T11:20:22.699 INFO:teuthology.orchestra.run.smithi139.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-09-06T11:20:22.699 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-09-06T11:20:22.699 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 x86_64 2:17.0.0-7472.g2227f238.el8 ceph 482 k 2021-09-06T11:20:22.699 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-09-06T11:20:22.700 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2 x86_64 2:17.0.0-7472.g2227f238.el8 ceph 3.7 M 2021-09-06T11:20:22.700 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-09-06T11:20:22.700 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-09-06T11:20:22.700 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse 2021-09-06T11:20:22.701 INFO:teuthology.orchestra.run.smithi139.stdout: x86_64 2:17.0.0-7472.g2227f238.el8 ceph 45 k 2021-09-06T11:20:22.701 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common x86_64 2:17.0.0-7472.g2227f238.el8 ceph 78 k 2021-09-06T11:20:22.701 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs x86_64 2:17.0.0-7472.g2227f238.el8 ceph 178 k 2021-09-06T11:20:22.701 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-09-06T11:20:22.702 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados x86_64 2:17.0.0-7472.g2227f238.el8 ceph 365 k 2021-09-06T11:20:22.702 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd x86_64 2:17.0.0-7472.g2227f238.el8 ceph 346 k 2021-09-06T11:20:22.702 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw x86_64 2:17.0.0-7472.g2227f238.el8 ceph 110 k 2021-09-06T11:20:22.702 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:20:22.702 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:20:22.703 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:20:22.703 INFO:teuthology.orchestra.run.smithi139.stdout:Install 21 Packages 2021-09-06T11:20:22.703 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 2 Packages 2021-09-06T11:20:22.703 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:20:22.704 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 48 M 2021-09-06T11:20:22.704 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:20:23.885 INFO:teuthology.orchestra.run.smithi139.stdout:(1/23): ceph-common-17.0.0-7472.g2227f238.el8.x 17 MB/s | 20 MB 00:01 2021-09-06T11:20:23.906 INFO:teuthology.orchestra.run.smithi139.stdout:(2/23): ceph-selinux-17.0.0-7472.g2227f238.el8. 1.1 MB/s | 24 kB 00:00 2021-09-06T11:20:23.945 INFO:teuthology.orchestra.run.smithi139.stdout:(3/23): libcephfs2-17.0.0-7472.g2227f238.el8.x8 20 MB/s | 783 kB 00:00 2021-09-06T11:20:23.959 INFO:teuthology.orchestra.run.smithi139.stdout:(4/23): ceph-base-17.0.0-7472.g2227f238.el8.x86 4.1 MB/s | 5.2 MB 00:01 2021-09-06T11:20:24.069 INFO:teuthology.orchestra.run.smithi139.stdout:(5/23): libradosstriper1-17.0.0-7472.g2227f238. 3.8 MB/s | 482 kB 00:00 2021-09-06T11:20:24.125 INFO:teuthology.orchestra.run.smithi139.stdout:(6/23): librgw2-17.0.0-7472.g2227f238.el8.x86_6 23 MB/s | 3.7 MB 00:00 2021-09-06T11:20:24.130 INFO:teuthology.orchestra.run.smithi139.stdout:(7/23): python3-ceph-argparse-17.0.0-7472.g2227 723 kB/s | 45 kB 00:00 2021-09-06T11:20:24.146 INFO:teuthology.orchestra.run.smithi139.stdout:(8/23): python3-ceph-common-17.0.0-7472.g2227f2 3.6 MB/s | 78 kB 00:00 2021-09-06T11:20:24.168 INFO:teuthology.orchestra.run.smithi139.stdout:(9/23): python3-cephfs-17.0.0-7472.g2227f238.el 4.6 MB/s | 178 kB 00:00 2021-09-06T11:20:24.170 INFO:teuthology.orchestra.run.smithi139.stdout:(10/23): python3-rados-17.0.0-7472.g2227f238.el 15 MB/s | 365 kB 00:00 2021-09-06T11:20:24.192 INFO:teuthology.orchestra.run.smithi139.stdout:(11/23): python3-rgw-17.0.0-7472.g2227f238.el8. 5.0 MB/s | 110 kB 00:00 2021-09-06T11:20:24.226 INFO:teuthology.orchestra.run.smithi139.stdout:(12/23): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2021-09-06T11:20:24.230 INFO:teuthology.orchestra.run.smithi139.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 65 MB/s | 259 kB 00:00 2021-09-06T11:20:24.232 INFO:teuthology.orchestra.run.smithi139.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 29 MB/s | 44 kB 00:00 2021-09-06T11:20:24.233 INFO:teuthology.orchestra.run.smithi139.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 36 MB/s | 47 kB 00:00 2021-09-06T11:20:24.253 INFO:teuthology.orchestra.run.smithi139.stdout:(16/23): python3-rbd-17.0.0-7472.g2227f238.el8. 4.0 MB/s | 346 kB 00:00 2021-09-06T11:20:24.497 INFO:teuthology.orchestra.run.smithi139.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 741 kB/s | 181 kB 00:00 2021-09-06T11:20:24.510 INFO:teuthology.orchestra.run.smithi139.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2021-09-06T11:20:24.546 INFO:teuthology.orchestra.run.smithi139.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2021-09-06T11:20:24.554 INFO:teuthology.orchestra.run.smithi139.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.7 MB/s | 75 kB 00:00 2021-09-06T11:20:24.726 INFO:teuthology.orchestra.run.smithi139.stdout:(21/23): librbd1-17.0.0-7472.g2227f238.el8.x86_ 22 MB/s | 3.9 MB 00:00 2021-09-06T11:20:25.192 INFO:teuthology.orchestra.run.smithi139.stdout:(22/23): librados2-17.0.0-7472.g2227f238.el8.x8 5.3 MB/s | 3.4 MB 00:00 2021-09-06T11:20:25.558 INFO:teuthology.orchestra.run.smithi139.stdout:(23/23): ceph-radosgw-17.0.0-7472.g2227f238.el8 2.9 MB/s | 8.4 MB 00:02 2021-09-06T11:20:25.561 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:20:25.562 INFO:teuthology.orchestra.run.smithi139.stdout:Total 17 MB/s | 48 MB 00:02 2021-09-06T11:20:25.566 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:20:25.611 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:20:25.611 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:20:26.015 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:20:26.018 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:20:27.022 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:20:27.199 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2021-09-06T11:20:27.355 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-09-06T11:20:27.793 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-09-06T11:20:27.988 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librados2-2:17.0.0-7472.g2227f238.el8.x86_64 3/25 2021-09-06T11:20:28.082 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:17.0.0-7472.g2227f238.el8.x86_64 3/25 2021-09-06T11:20:28.286 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-2:17.0.0-7472.g2227f238.el8.x86_64 4/25 2021-09-06T11:20:28.472 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-2:17.0.0-7472.g2227f238.el8.x86_64 5/25 2021-09-06T11:20:28.890 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-2:17.0.0-7472.g2227f238.el8.x86_64 5/25 2021-09-06T11:20:29.046 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librbd1-2:17.0.0-7472.g2227f238.el8.x86_64 6/25 2021-09-06T11:20:29.094 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:17.0.0-7472.g2227f238.el8.x86_64 6/25 2021-09-06T11:20:29.245 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-09-06T11:20:29.462 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-06T11:20:29.862 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-06T11:20:30.046 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librgw2-2:17.0.0-7472.g2227f238.el8.x86_64 9/25 2021-09-06T11:20:30.110 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librgw2-2:17.0.0-7472.g2227f238.el8.x86_64 9/25 2021-09-06T11:20:30.277 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-09-06T11:20:30.423 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-argparse-2:17.0.0-7472.g2227f238.el8. 11/25 2021-09-06T11:20:30.583 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-2:17.0.0-7472.g2227f238.el8.x86_64 12/25 2021-09-06T11:20:30.822 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-2:17.0.0-7472.g2227f238.el8.x86_64 13/25 2021-09-06T11:20:30.998 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-2:17.0.0-7472.g2227f238.el8.x86_64 14/25 2021-09-06T11:20:31.179 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libradosstriper1-2:17.0.0-7472.g2227f238.el8.x86_6 15/25 2021-09-06T11:20:31.232 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7472.g2227f238.el8.x86_6 15/25 2021-09-06T11:20:31.397 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2021-09-06T11:20:31.630 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2021-09-06T11:20:31.792 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2021-09-06T11:20:31.929 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-common-2:17.0.0-7472.g2227f238.el8.x8 19/25 2021-09-06T11:20:35.906 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 20/25 2021-09-06T11:20:36.079 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 20/25 2021-09-06T11:20:36.650 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 20/25 2021-09-06T11:20:36.897 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-2:17.0.0-7472.g2227f238.el8.x86_64 21/25 2021-09-06T11:20:37.002 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-2:17.0.0-7472.g2227f238.el8.x86_64 21/25 2021-09-06T11:20:37.095 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-selinux-2:17.0.0-7472.g2227f238.el8.x86_64 22/25 2021-09-06T11:20:52.606 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7472.g2227f238.el8.x86_64 22/25 2021-09-06T11:20:52.607 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /sys 2021-09-06T11:20:52.607 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /proc 2021-09-06T11:20:52.608 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /dev 2021-09-06T11:20:52.608 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /run 2021-09-06T11:20:52.608 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /mnt 2021-09-06T11:20:52.608 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /var/tmp 2021-09-06T11:20:52.608 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /home 2021-09-06T11:20:52.609 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /tmp 2021-09-06T11:20:52.609 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /dev 2021-09-06T11:20:52.609 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:20:53.686 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-2:17.0.0-7472.g2227f238.el8.x86_64 23/25 2021-09-06T11:20:53.762 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7472.g2227f238.el8.x86_64 23/25 2021-09-06T11:20:53.762 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-06T11:20:53.944 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-06T11:20:53.945 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T11:20:54.701 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T11:20:54.702 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-2:17.0.0-7472.g2227f238.el8.x86_64 1/25 2021-09-06T11:20:54.702 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 2/25 2021-09-06T11:20:54.702 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-2:17.0.0-7472.g2227f238.el8.x86_64 3/25 2021-09-06T11:20:54.703 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-selinux-2:17.0.0-7472.g2227f238.el8.x86_64 4/25 2021-09-06T11:20:54.703 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-2:17.0.0-7472.g2227f238.el8.x86_64 5/25 2021-09-06T11:20:54.703 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libradosstriper1-2:17.0.0-7472.g2227f238.el8.x86_6 6/25 2021-09-06T11:20:54.703 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librgw2-2:17.0.0-7472.g2227f238.el8.x86_64 7/25 2021-09-06T11:20:54.704 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7472.g2227f238.el8. 8/25 2021-09-06T11:20:54.704 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-common-2:17.0.0-7472.g2227f238.el8.x8 9/25 2021-09-06T11:20:54.704 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-2:17.0.0-7472.g2227f238.el8.x86_64 10/25 2021-09-06T11:20:54.704 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-2:17.0.0-7472.g2227f238.el8.x86_64 11/25 2021-09-06T11:20:54.704 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-2:17.0.0-7472.g2227f238.el8.x86_64 12/25 2021-09-06T11:20:54.705 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-2:17.0.0-7472.g2227f238.el8.x86_64 13/25 2021-09-06T11:20:54.705 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2021-09-06T11:20:54.706 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2021-09-06T11:20:54.706 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-09-06T11:20:54.706 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2021-09-06T11:20:54.707 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-09-06T11:20:54.707 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2021-09-06T11:20:54.708 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2021-09-06T11:20:54.708 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2021-09-06T11:20:54.708 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:17.0.0-7472.g2227f238.el8.x86_64 22/25 2021-09-06T11:20:54.708 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-09-06T11:20:54.708 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:17.0.0-7472.g2227f238.el8.x86_64 24/25 2021-09-06T11:20:55.371 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T11:20:55.371 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:20:55.371 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2021-09-06T11:20:55.372 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.372 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.372 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:20:55.373 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:20:55.373 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.373 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.373 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.374 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.374 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-09-06T11:20:55.374 INFO:teuthology.orchestra.run.smithi139.stdout: leveldb-1.22-1.el8.x86_64 2021-09-06T11:20:55.374 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.374 INFO:teuthology.orchestra.run.smithi139.stdout: liboath-2.6.2-3.el8.x86_64 2021-09-06T11:20:55.375 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-09-06T11:20:55.375 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.375 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-09-06T11:20:55.376 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.376 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind-1.3.1-3.el8.x86_64 2021-09-06T11:20:55.376 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-09-06T11:20:55.376 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.376 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.377 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.377 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-09-06T11:20:55.377 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.377 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.378 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:20:55.378 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:20:55.378 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:20:55.603 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-test 2021-09-06T11:20:56.058 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:36 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:20:57.022 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:20:57.023 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:20:57.024 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-09-06T11:20:57.024 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:20:57.024 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:20:57.024 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test x86_64 2:17.0.0-7472.g2227f238.el8 ceph 43 M 2021-09-06T11:20:57.024 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-09-06T11:20:57.025 INFO:teuthology.orchestra.run.smithi139.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-09-06T11:20:57.025 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite x86_64 2:17.0.0-7472.g2227f238.el8 ceph 170 k 2021-09-06T11:20:57.025 INFO:teuthology.orchestra.run.smithi139.stdout: libxslt x86_64 1.1.32-6.el8 CentOS-Base 250 k 2021-09-06T11:20:57.025 INFO:teuthology.orchestra.run.smithi139.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-09-06T11:20:57.026 INFO:teuthology.orchestra.run.smithi139.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-09-06T11:20:57.026 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-09-06T11:20:57.026 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:20:57.026 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:20:57.027 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:20:57.027 INFO:teuthology.orchestra.run.smithi139.stdout:Install 7 Packages 2021-09-06T11:20:57.027 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:20:57.028 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 44 M 2021-09-06T11:20:57.028 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 207 M 2021-09-06T11:20:57.028 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:20:57.060 INFO:teuthology.orchestra.run.smithi139.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 4.7 MB/s | 161 kB 00:00 2021-09-06T11:20:57.063 INFO:teuthology.orchestra.run.smithi139.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 59 MB/s | 187 kB 00:00 2021-09-06T11:20:57.067 INFO:teuthology.orchestra.run.smithi139.stdout:(3/7): socat-1.7.3.3-2.el8.x86_64.rpm 70 MB/s | 302 kB 00:00 2021-09-06T11:20:57.071 INFO:teuthology.orchestra.run.smithi139.stdout:(4/7): libxslt-1.1.32-6.el8.x86_64.rpm 65 MB/s | 250 kB 00:00 2021-09-06T11:20:57.411 INFO:teuthology.orchestra.run.smithi139.stdout:(5/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 203 kB/s | 69 kB 00:00 2021-09-06T11:20:57.468 INFO:teuthology.orchestra.run.smithi139.stdout:(6/7): libcephsqlite-17.0.0-7472.g2227f238.el8. 385 kB/s | 170 kB 00:00 2021-09-06T11:20:59.516 INFO:teuthology.orchestra.run.smithi139.stdout:(7/7): ceph-test-17.0.0-7472.g2227f238.el8.x86_ 17 MB/s | 43 MB 00:02 2021-09-06T11:20:59.521 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:20:59.521 INFO:teuthology.orchestra.run.smithi139.stdout:Total 18 MB/s | 44 MB 00:02 2021-09-06T11:20:59.525 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:20:59.549 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:20:59.549 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:20:59.792 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:20:59.795 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:21:00.271 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:21:00.449 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libxslt-1.1.32-6.el8.x86_64 1/7 2021-09-06T11:21:00.652 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-09-06T11:21:00.844 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-09-06T11:21:01.716 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-06T11:21:01.792 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-06T11:21:01.948 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-09-06T11:21:02.165 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephsqlite-2:17.0.0-7472.g2227f238.el8.x86_64 6/7 2021-09-06T11:21:06.426 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7472.g2227f238.el8.x86_64 6/7 2021-09-06T11:21:06.704 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-2:17.0.0-7472.g2227f238.el8.x86_64 7/7 2021-09-06T11:21:06.947 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-test-2:17.0.0-7472.g2227f238.el8.x86_64 7/7 2021-09-06T11:21:06.947 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-2:17.0.0-7472.g2227f238.el8.x86_64 1/7 2021-09-06T11:21:06.947 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephsqlite-2:17.0.0-7472.g2227f238.el8.x86_64 2/7 2021-09-06T11:21:06.948 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2021-09-06T11:21:06.948 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-06T11:21:06.948 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/7 2021-09-06T11:21:06.948 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libxslt-1.1.32-6.el8.x86_64 6/7 2021-09-06T11:21:07.751 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2021-09-06T11:21:07.751 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:07.751 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:21:07.752 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-2:17.0.0-7472.g2227f238.el8.x86_64 jq-1.5-12.el8.x86_64 2021-09-06T11:21:07.752 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite-2:17.0.0-7472.g2227f238.el8.x86_64 libxslt-1.1.32-6.el8.x86_64 2021-09-06T11:21:07.752 INFO:teuthology.orchestra.run.smithi139.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-09-06T11:21:07.752 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-09-06T11:21:07.753 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:07.753 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:21:08.986 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph 2021-09-06T11:21:09.437 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:49 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:21:10.384 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:21:10.388 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2021-09-06T11:21:10.388 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-09-06T11:21:10.388 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2021-09-06T11:21:10.388 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:21:10.389 INFO:teuthology.orchestra.run.smithi139.stdout: ceph x86_64 2:17.0.0-7472.g2227f238.el8 ceph 6.4 k 2021-09-06T11:21:10.389 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-09-06T11:21:10.389 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds x86_64 2:17.0.0-7472.g2227f238.el8 ceph 2.3 M 2021-09-06T11:21:10.389 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr x86_64 2:17.0.0-7472.g2227f238.el8 ceph 1.5 M 2021-09-06T11:21:10.390 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 214 k 2021-09-06T11:21:10.390 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon x86_64 2:17.0.0-7472.g2227f238.el8 ceph 3.9 M 2021-09-06T11:21:10.390 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd x86_64 2:17.0.0-7472.g2227f238.el8 ceph 17 M 2021-09-06T11:21:10.390 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-09-06T11:21:10.391 INFO:teuthology.orchestra.run.smithi139.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-09-06T11:21:10.391 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-09-06T11:21:10.391 INFO:teuthology.orchestra.run.smithi139.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-09-06T11:21:10.391 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-09-06T11:21:10.391 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-09-06T11:21:10.392 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography x86_64 3.2.1-4.el8 CentOS-Base 559 k 2021-09-06T11:21:10.392 INFO:teuthology.orchestra.run.smithi139.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-09-06T11:21:10.392 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-09-06T11:21:10.392 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-09-06T11:21:10.392 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-09-06T11:21:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: python3-lxml x86_64 4.2.3-2.el8 CentOS-AppStream 1.5 M 2021-09-06T11:21:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-09-06T11:21:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-09-06T11:21:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-09-06T11:21:10.393 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-09-06T11:21:10.394 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-09-06T11:21:10.394 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-09-06T11:21:10.394 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-09-06T11:21:10.394 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-09-06T11:21:10.395 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-09-06T11:21:10.395 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-09-06T11:21:10.395 INFO:teuthology.orchestra.run.smithi139.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-09-06T11:21:10.395 INFO:teuthology.orchestra.run.smithi139.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-09-06T11:21:10.395 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-09-06T11:21:10.396 INFO:teuthology.orchestra.run.smithi139.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-09-06T11:21:10.396 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-09-06T11:21:10.396 INFO:teuthology.orchestra.run.smithi139.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-09-06T11:21:10.396 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-09-06T11:21:10.396 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-09-06T11:21:10.397 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-09-06T11:21:10.397 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-09-06T11:21:10.397 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-09-06T11:21:10.397 INFO:teuthology.orchestra.run.smithi139.stdout:Installing weak dependencies: 2021-09-06T11:21:10.398 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-09-06T11:21:10.398 INFO:teuthology.orchestra.run.smithi139.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-09-06T11:21:10.398 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:10.398 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:21:10.399 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2021-09-06T11:21:10.399 INFO:teuthology.orchestra.run.smithi139.stdout:Install 41 Packages 2021-09-06T11:21:10.399 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:10.400 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 31 M 2021-09-06T11:21:10.401 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 121 M 2021-09-06T11:21:10.401 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:21:10.602 INFO:teuthology.orchestra.run.smithi139.stdout:(1/41): ceph-17.0.0-7472.g2227f238.el8.x86_64.r 31 kB/s | 6.4 kB 00:00 2021-09-06T11:21:10.710 INFO:teuthology.orchestra.run.smithi139.stdout:(2/41): ceph-mgr-17.0.0-7472.g2227f238.el8.x86_ 4.8 MB/s | 1.5 MB 00:00 2021-09-06T11:21:11.478 INFO:teuthology.orchestra.run.smithi139.stdout:(3/41): ceph-mds-17.0.0-7472.g2227f238.el8.x86_ 2.1 MB/s | 2.3 MB 00:01 2021-09-06T11:21:11.666 INFO:teuthology.orchestra.run.smithi139.stdout:(4/41): ceph-osd-17.0.0-7472.g2227f238.el8.x86_ 18 MB/s | 17 MB 00:00 2021-09-06T11:21:11.667 INFO:teuthology.orchestra.run.smithi139.stdout:(5/41): ceph-mgr-modules-core-17.0.0-7472.g2227 1.1 MB/s | 214 kB 00:00 2021-09-06T11:21:11.712 INFO:teuthology.orchestra.run.smithi139.stdout:(6/41): python3-html5lib-0.999999999-6.el8.noar 4.5 MB/s | 214 kB 00:00 2021-09-06T11:21:11.716 INFO:teuthology.orchestra.run.smithi139.stdout:(7/41): python3-mako-1.0.6-13.el8.noarch.rpm 35 MB/s | 157 kB 00:00 2021-09-06T11:21:11.718 INFO:teuthology.orchestra.run.smithi139.stdout:(8/41): python3-markupsafe-0.23-19.el8.x86_64.r 25 MB/s | 39 kB 00:00 2021-09-06T11:21:11.722 INFO:teuthology.orchestra.run.smithi139.stdout:(9/41): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 29 MB/s | 103 kB 00:00 2021-09-06T11:21:11.725 INFO:teuthology.orchestra.run.smithi139.stdout:(10/41): python3-pytz-2017.2-9.el8.noarch.rpm 21 MB/s | 54 kB 00:00 2021-09-06T11:21:11.726 INFO:teuthology.orchestra.run.smithi139.stdout:(11/41): python3-webencodings-0.5.1-6.el8.noarc 15 MB/s | 27 kB 00:00 2021-09-06T11:21:11.736 INFO:teuthology.orchestra.run.smithi139.stdout:(12/41): python3-lxml-4.2.3-2.el8.x86_64.rpm 22 MB/s | 1.5 MB 00:00 2021-09-06T11:21:11.741 INFO:teuthology.orchestra.run.smithi139.stdout:(13/41): python3-werkzeug-0.12.2-4.el8.noarch.r 31 MB/s | 457 kB 00:00 2021-09-06T11:21:11.743 INFO:teuthology.orchestra.run.smithi139.stdout:(14/41): python3-cffi-1.11.5-5.el8.x86_64.rpm 40 MB/s | 237 kB 00:00 2021-09-06T11:21:11.745 INFO:teuthology.orchestra.run.smithi139.stdout:(15/41): python3-chardet-3.0.4-7.el8.noarch.rpm 52 MB/s | 195 kB 00:00 2021-09-06T11:21:11.753 INFO:teuthology.orchestra.run.smithi139.stdout:(16/41): python3-idna-2.5-5.el8.noarch.rpm 11 MB/s | 97 kB 00:00 2021-09-06T11:21:11.756 INFO:teuthology.orchestra.run.smithi139.stdout:(17/41): python3-cryptography-3.2.1-4.el8.x86_6 40 MB/s | 559 kB 00:00 2021-09-06T11:21:11.757 INFO:teuthology.orchestra.run.smithi139.stdout:(18/41): python3-pycparser-2.14-14.el8.noarch.r 31 MB/s | 109 kB 00:00 2021-09-06T11:21:11.757 INFO:teuthology.orchestra.run.smithi139.stdout:(19/41): python3-pysocks-1.6.8-3.el8.noarch.rpm 21 MB/s | 34 kB 00:00 2021-09-06T11:21:11.760 INFO:teuthology.orchestra.run.smithi139.stdout:(20/41): python3-requests-2.20.0-2.1.el8_1.noar 49 MB/s | 123 kB 00:00 2021-09-06T11:21:11.761 INFO:teuthology.orchestra.run.smithi139.stdout:(21/41): python3-urllib3-1.24.2-5.el8.noarch.rp 52 MB/s | 177 kB 00:00 2021-09-06T11:21:11.977 INFO:teuthology.orchestra.run.smithi139.stdout:(22/41): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 200 kB/s | 44 kB 00:00 2021-09-06T11:21:12.029 INFO:teuthology.orchestra.run.smithi139.stdout:(23/41): python3-beautifulsoup4-4.6.3-2.el8.1.n 692 kB/s | 185 kB 00:00 2021-09-06T11:21:12.064 INFO:teuthology.orchestra.run.smithi139.stdout:(24/41): ceph-mon-17.0.0-7472.g2227f238.el8.x86 2.6 MB/s | 3.9 MB 00:01 2021-09-06T11:21:12.133 INFO:teuthology.orchestra.run.smithi139.stdout:(25/41): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2021-09-06T11:21:12.173 INFO:teuthology.orchestra.run.smithi139.stdout:(26/41): python3-jaraco-6.2-6.el8.noarch.rpm 278 kB/s | 11 kB 00:00 2021-09-06T11:21:12.175 INFO:teuthology.orchestra.run.smithi139.stdout:(27/41): python3-cherrypy-18.4.0-1.el8.noarch.r 2.6 MB/s | 384 kB 00:00 2021-09-06T11:21:12.208 INFO:teuthology.orchestra.run.smithi139.stdout:(28/41): python3-jaraco-functools-2.0-4.el8.noa 508 kB/s | 18 kB 00:00 2021-09-06T11:21:12.208 INFO:teuthology.orchestra.run.smithi139.stdout:(29/41): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2021-09-06T11:21:12.213 INFO:teuthology.orchestra.run.smithi139.stdout:(30/41): python3-cssselect-0.9.2-10.el8.noarch. 270 kB/s | 40 kB 00:00 2021-09-06T11:21:12.240 INFO:teuthology.orchestra.run.smithi139.stdout:(31/41): python3-more-itertools-7.2.0-3.el8.noa 1.8 MB/s | 59 kB 00:00 2021-09-06T11:21:12.250 INFO:teuthology.orchestra.run.smithi139.stdout:(32/41): python3-portend-2.6-1.el8.noarch.rpm 419 kB/s | 16 kB 00:00 2021-09-06T11:21:12.282 INFO:teuthology.orchestra.run.smithi139.stdout:(33/41): python3-simplegeneric-0.8.1-17.el8.noa 440 kB/s | 19 kB 00:00 2021-09-06T11:21:12.284 INFO:teuthology.orchestra.run.smithi139.stdout:(34/41): python3-pecan-1.3.2-9.el8.noarch.rpm 3.6 MB/s | 283 kB 00:00 2021-09-06T11:21:12.297 INFO:teuthology.orchestra.run.smithi139.stdout:(35/41): python3-singledispatch-3.4.0.3-18.el8. 510 kB/s | 24 kB 00:00 2021-09-06T11:21:12.330 INFO:teuthology.orchestra.run.smithi139.stdout:(36/41): python3-tempora-1.14.1-5.el8.noarch.rp 617 kB/s | 29 kB 00:00 2021-09-06T11:21:12.333 INFO:teuthology.orchestra.run.smithi139.stdout:(37/41): python3-trustme-0.6.0-4.el8.noarch.rpm 574 kB/s | 27 kB 00:00 2021-09-06T11:21:12.390 INFO:teuthology.orchestra.run.smithi139.stdout:(38/41): python3-webtest-2.0.33-1.el8.noarch.rp 1.5 MB/s | 85 kB 00:00 2021-09-06T11:21:12.438 INFO:teuthology.orchestra.run.smithi139.stdout:(39/41): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2021-09-06T11:21:12.444 INFO:teuthology.orchestra.run.smithi139.stdout:(40/41): python3-zc-lockfile-2.0-2.el8.noarch.r 426 kB/s | 23 kB 00:00 2021-09-06T11:21:12.466 INFO:teuthology.orchestra.run.smithi139.stdout:(41/41): python3-waitress-1.2.1-2.el8.1.noarch. 1.4 MB/s | 241 kB 00:00 2021-09-06T11:21:12.470 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:21:12.471 INFO:teuthology.orchestra.run.smithi139.stdout:Total 15 MB/s | 31 MB 00:02 2021-09-06T11:21:12.484 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:21:12.525 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:21:12.525 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:21:12.823 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:21:12.826 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:21:14.322 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:21:14.547 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/41 2021-09-06T11:21:14.750 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/41 2021-09-06T11:21:14.947 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-idna-2.5-5.el8.noarch 3/41 2021-09-06T11:21:15.162 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/41 2021-09-06T11:21:15.378 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/41 2021-09-06T11:21:15.518 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/41 2021-09-06T11:21:15.657 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/41 2021-09-06T11:21:15.874 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/41 2021-09-06T11:21:16.019 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/41 2021-09-06T11:21:16.199 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/41 2021-09-06T11:21:16.362 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/41 2021-09-06T11:21:16.610 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/41 2021-09-06T11:21:16.861 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 13/41 2021-09-06T11:21:17.092 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/41 2021-09-06T11:21:17.454 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-09-06T11:21:17.717 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 16/41 2021-09-06T11:21:17.887 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 17/41 2021-09-06T11:21:18.131 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 18/41 2021-09-06T11:21:18.314 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 19/41 2021-09-06T11:21:18.588 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 20/41 2021-09-06T11:21:18.793 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 21/41 2021-09-06T11:21:19.076 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 22/41 2021-09-06T11:21:19.299 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/41 2021-09-06T11:21:19.551 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 24/41 2021-09-06T11:21:19.751 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 25/41 2021-09-06T11:21:20.161 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 26/41 2021-09-06T11:21:20.448 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-lxml-4.2.3-2.el8.x86_64 27/41 2021-09-06T11:21:20.652 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 28/41 2021-09-06T11:21:20.866 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 29/41 2021-09-06T11:21:21.017 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 30/41 2021-09-06T11:21:21.264 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-portend-2.6-1.el8.noarch 31/41 2021-09-06T11:21:21.466 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 32/41 2021-09-06T11:21:21.684 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/41 2021-09-06T11:21:22.070 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/41 2021-09-06T11:21:22.411 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 35/41 2021-09-06T11:21:22.724 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7472.g2227f238.el8. 36/41 2021-09-06T11:21:22.897 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-2:17.0.0-7472.g2227f238.el8.x86_64 37/41 2021-09-06T11:21:24.502 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7472.g2227f238.el8.x86_64 37/41 2021-09-06T11:21:25.428 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-2:17.0.0-7472.g2227f238.el8.x86_64 38/41 2021-09-06T11:21:25.823 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-2:17.0.0-7472.g2227f238.el8.x86_64 38/41 2021-09-06T11:21:26.045 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-2:17.0.0-7472.g2227f238.el8.x86_64 39/41 2021-09-06T11:21:26.280 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-2:17.0.0-7472.g2227f238.el8.x86_64 39/41 2021-09-06T11:21:26.436 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-2:17.0.0-7472.g2227f238.el8.x86_64 40/41 2021-09-06T11:21:26.453 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-2:17.0.0-7472.g2227f238.el8.x86_64 40/41 2021-09-06T11:21:26.602 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-2:17.0.0-7472.g2227f238.el8.x86_64 41/41 2021-09-06T11:21:26.978 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-2:17.0.0-7472.g2227f238.el8.x86_64 41/41 2021-09-06T11:21:26.979 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-2:17.0.0-7472.g2227f238.el8.x86_64 1/41 2021-09-06T11:21:26.979 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-2:17.0.0-7472.g2227f238.el8.x86_64 2/41 2021-09-06T11:21:26.979 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-2:17.0.0-7472.g2227f238.el8.x86_64 3/41 2021-09-06T11:21:26.979 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-2:17.0.0-7472.g2227f238.el8.x86_64 4/41 2021-09-06T11:21:26.979 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-2:17.0.0-7472.g2227f238.el8.x86_64 5/41 2021-09-06T11:21:26.980 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7472.g2227f238.el8. 6/41 2021-09-06T11:21:26.980 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/41 2021-09-06T11:21:26.980 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-lxml-4.2.3-2.el8.x86_64 8/41 2021-09-06T11:21:26.980 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/41 2021-09-06T11:21:26.981 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/41 2021-09-06T11:21:26.981 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 11/41 2021-09-06T11:21:26.981 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/41 2021-09-06T11:21:26.982 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/41 2021-09-06T11:21:26.982 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/41 2021-09-06T11:21:26.982 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-09-06T11:21:26.983 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 16/41 2021-09-06T11:21:26.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 17/41 2021-09-06T11:21:26.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-idna-2.5-5.el8.noarch 18/41 2021-09-06T11:21:26.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 19/41 2021-09-06T11:21:26.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 20/41 2021-09-06T11:21:26.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 21/41 2021-09-06T11:21:26.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 22/41 2021-09-06T11:21:26.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 23/41 2021-09-06T11:21:26.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/41 2021-09-06T11:21:26.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 25/41 2021-09-06T11:21:26.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 26/41 2021-09-06T11:21:26.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 27/41 2021-09-06T11:21:26.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 28/41 2021-09-06T11:21:26.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 29/41 2021-09-06T11:21:26.987 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 30/41 2021-09-06T11:21:26.987 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 31/41 2021-09-06T11:21:26.987 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 32/41 2021-09-06T11:21:26.987 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-portend-2.6-1.el8.noarch 33/41 2021-09-06T11:21:26.987 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 34/41 2021-09-06T11:21:26.988 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 35/41 2021-09-06T11:21:26.988 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 36/41 2021-09-06T11:21:26.988 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 37/41 2021-09-06T11:21:26.988 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 38/41 2021-09-06T11:21:26.989 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 39/41 2021-09-06T11:21:26.989 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 40/41 2021-09-06T11:21:27.879 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 41/41 2021-09-06T11:21:27.879 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:27.880 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:21:27.880 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:21:27.880 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:21:27.880 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:21:27.881 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:21:27.881 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:21:27.881 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:21:27.881 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-09-06T11:21:27.881 INFO:teuthology.orchestra.run.smithi139.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-09-06T11:21:27.882 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-09-06T11:21:27.882 INFO:teuthology.orchestra.run.smithi139.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-09-06T11:21:27.882 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-09-06T11:21:27.882 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-09-06T11:21:27.883 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-09-06T11:21:27.883 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-09-06T11:21:27.883 INFO:teuthology.orchestra.run.smithi139.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-09-06T11:21:27.883 INFO:teuthology.orchestra.run.smithi139.stdout: python3-idna-2.5-5.el8.noarch 2021-09-06T11:21:27.884 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-6.2-6.el8.noarch 2021-09-06T11:21:27.884 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-09-06T11:21:27.884 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-09-06T11:21:27.884 INFO:teuthology.orchestra.run.smithi139.stdout: python3-lxml-4.2.3-2.el8.x86_64 2021-09-06T11:21:27.885 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako-1.0.6-13.el8.noarch 2021-09-06T11:21:27.885 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-09-06T11:21:27.885 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-09-06T11:21:27.885 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-09-06T11:21:27.885 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend-2.6-1.el8.noarch 2021-09-06T11:21:27.886 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-09-06T11:21:27.886 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser-2.14-14.el8.noarch 2021-09-06T11:21:27.886 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-09-06T11:21:27.886 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz-2017.2-9.el8.noarch 2021-09-06T11:21:27.887 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-09-06T11:21:27.887 INFO:teuthology.orchestra.run.smithi139.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-09-06T11:21:27.887 INFO:teuthology.orchestra.run.smithi139.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-09-06T11:21:27.887 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-09-06T11:21:27.887 INFO:teuthology.orchestra.run.smithi139.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-09-06T11:21:27.888 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-09-06T11:21:27.888 INFO:teuthology.orchestra.run.smithi139.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-09-06T11:21:27.888 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-09-06T11:21:27.888 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-09-06T11:21:27.888 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-09-06T11:21:27.889 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-09-06T11:21:27.889 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-09-06T11:21:27.889 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:27.890 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:21:28.085 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-base 2021-09-06T11:21:28.536 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:08 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:21:29.234 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-base-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:21:29.486 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:21:29.487 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-09-06T11:21:29.487 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:21:29.531 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install cephadm 2021-09-06T11:21:29.971 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:09 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:21:31.028 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:21:31.029 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:21:31.030 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-09-06T11:21:31.030 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:21:31.030 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:21:31.030 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 69 k 2021-09-06T11:21:31.031 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:31.031 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:21:31.031 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:21:31.031 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-09-06T11:21:31.032 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:31.032 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 69 k 2021-09-06T11:21:31.032 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 309 k 2021-09-06T11:21:31.032 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:21:31.200 INFO:teuthology.orchestra.run.smithi139.stdout:cephadm-17.0.0-7472.g2227f238.el8.noarch.rpm 409 kB/s | 69 kB 00:00 2021-09-06T11:21:31.201 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:21:31.201 INFO:teuthology.orchestra.run.smithi139.stdout:Total 405 kB/s | 69 kB 00:00 2021-09-06T11:21:31.202 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:21:31.208 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:21:31.208 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:21:31.210 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:21:31.213 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:21:31.265 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:21:33.451 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:17.0.0-7472.g2227f238.el8.noarch 1/1 2021-09-06T11:21:33.716 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephadm-2:17.0.0-7472.g2227f238.el8.noarch 1/1 2021-09-06T11:21:33.897 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:17.0.0-7472.g2227f238.el8.noarch 1/1 2021-09-06T11:21:34.225 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephadm-2:17.0.0-7472.g2227f238.el8.noarch 1/1 2021-09-06T11:21:34.225 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:34.225 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:21:34.225 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:21:34.226 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:34.226 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:21:34.360 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-immutable-object-cache 2021-09-06T11:21:34.820 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:14 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:21:35.776 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:21:35.778 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:21:35.778 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2021-09-06T11:21:35.778 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:21:35.778 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:21:35.779 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7472.g2227f238.el8 ceph 169 k 2021-09-06T11:21:35.779 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:35.779 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:21:35.779 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:21:35.779 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-09-06T11:21:35.780 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:35.780 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 169 k 2021-09-06T11:21:35.780 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 567 k 2021-09-06T11:21:35.780 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:21:36.000 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-immutable-object-cache-17.0.0-7472.g2227f2 763 kB/s | 169 kB 00:00 2021-09-06T11:21:36.000 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:21:36.001 INFO:teuthology.orchestra.run.smithi139.stdout:Total 757 kB/s | 169 kB 00:00 2021-09-06T11:21:36.001 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:21:36.013 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:21:36.014 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:21:36.063 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:21:36.066 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:21:36.938 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:21:37.685 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7472.g2227f238. 1/1 2021-09-06T11:21:38.047 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7472.g2227f238. 1/1 2021-09-06T11:21:38.299 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7472.g2227f238. 1/1 2021-09-06T11:21:38.300 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:38.300 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:21:38.300 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:21:38.301 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:38.301 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:21:38.477 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr 2021-09-06T11:21:38.925 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:18 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:21:39.641 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-mgr-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:21:39.893 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:21:39.894 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-09-06T11:21:39.894 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:21:39.955 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-dashboard 2021-09-06T11:21:40.398 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:20 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:21:41.362 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:21:41.362 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:21:41.363 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-09-06T11:21:41.363 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:21:41.363 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:21:41.364 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 1.5 M 2021-09-06T11:21:41.364 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-09-06T11:21:41.364 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 24 k 2021-09-06T11:21:41.364 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 8.9 k 2021-09-06T11:21:41.365 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-09-06T11:21:41.365 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-09-06T11:21:41.365 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-09-06T11:21:41.365 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:41.366 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:21:41.366 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:21:41.366 INFO:teuthology.orchestra.run.smithi139.stdout:Install 6 Packages 2021-09-06T11:21:41.367 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:41.367 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 1.8 M 2021-09-06T11:21:41.368 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 7.4 M 2021-09-06T11:21:41.368 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:21:41.465 INFO:teuthology.orchestra.run.smithi139.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7472.g2227 89 kB/s | 8.9 kB 00:00 2021-09-06T11:21:41.476 INFO:teuthology.orchestra.run.smithi139.stdout:(2/6): ceph-grafana-dashboards-17.0.0-7472.g222 214 kB/s | 24 kB 00:00 2021-09-06T11:21:41.501 INFO:teuthology.orchestra.run.smithi139.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.2 MB/s | 43 kB 00:00 2021-09-06T11:21:41.552 INFO:teuthology.orchestra.run.smithi139.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 451 kB/s | 34 kB 00:00 2021-09-06T11:21:41.608 INFO:teuthology.orchestra.run.smithi139.stdout:(5/6): ceph-mgr-dashboard-17.0.0-7472.g2227f238 6.3 MB/s | 1.5 MB 00:00 2021-09-06T11:21:41.642 INFO:teuthology.orchestra.run.smithi139.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2021-09-06T11:21:41.644 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:21:41.645 INFO:teuthology.orchestra.run.smithi139.stdout:Total 6.5 MB/s | 1.8 MB 00:00 2021-09-06T11:21:41.646 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:21:41.653 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:21:41.654 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:21:41.713 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:21:41.717 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:21:41.966 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:21:42.208 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-09-06T11:21:42.419 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-09-06T11:21:42.661 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-09-06T11:21:42.823 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7472.g2227f238.el8.n 4/6 2021-09-06T11:21:43.165 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7472.g2227f238.el8. 5/6 2021-09-06T11:21:43.368 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7472.g2227f238.el8.noarc 6/6 2021-09-06T11:21:43.589 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7472.g2227f238.el8.noarc 6/6 2021-09-06T11:21:43.589 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7472.g2227f238.el8. 1/6 2021-09-06T11:21:43.589 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7472.g2227f238.el8.noarc 2/6 2021-09-06T11:21:43.590 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7472.g2227f238.el8.n 3/6 2021-09-06T11:21:43.590 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-09-06T11:21:43.590 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-09-06T11:21:43.873 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-09-06T11:21:43.873 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:43.873 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:21:43.873 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:21:43.874 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:21:43.874 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:21:43.874 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-09-06T11:21:43.874 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-09-06T11:21:43.875 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes-2.4.1-12.el8.noarch 2021-09-06T11:21:43.875 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:43.875 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:21:43.972 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-diskprediction-local 2021-09-06T11:21:44.418 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:24 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:21:45.370 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:21:45.371 INFO:teuthology.orchestra.run.smithi139.stdout:===================================================================================================== 2021-09-06T11:21:45.372 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-09-06T11:21:45.372 INFO:teuthology.orchestra.run.smithi139.stdout:===================================================================================================== 2021-09-06T11:21:45.372 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:21:45.372 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 7.4 M 2021-09-06T11:21:45.373 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-09-06T11:21:45.373 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran x86_64 8.4.1-1.el8 CentOS-Base 639 k 2021-09-06T11:21:45.373 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath x86_64 8.4.1-1.el8 CentOS-Base 169 k 2021-09-06T11:21:45.373 INFO:teuthology.orchestra.run.smithi139.stdout: openblas x86_64 0.3.12-1.el8 CentOS-AppStream 4.6 M 2021-09-06T11:21:45.373 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-threads x86_64 0.3.12-1.el8 CentOS-AppStream 4.7 M 2021-09-06T11:21:45.374 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-09-06T11:21:45.374 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-09-06T11:21:45.374 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy x86_64 1.0.0-20.module_el8.4.0+790+083e3d81 CentOS-AppStream 15 M 2021-09-06T11:21:45.374 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:45.375 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:21:45.375 INFO:teuthology.orchestra.run.smithi139.stdout:===================================================================================================== 2021-09-06T11:21:45.375 INFO:teuthology.orchestra.run.smithi139.stdout:Install 8 Packages 2021-09-06T11:21:45.375 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:45.376 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 36 M 2021-09-06T11:21:45.376 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 197 M 2021-09-06T11:21:45.377 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:21:45.477 INFO:teuthology.orchestra.run.smithi139.stdout:(1/8): openblas-0.3.12-1.el8.x86_64.rpm 45 MB/s | 4.6 MB 00:00 2021-09-06T11:21:45.531 INFO:teuthology.orchestra.run.smithi139.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 69 MB/s | 3.7 MB 00:00 2021-09-06T11:21:45.536 INFO:teuthology.orchestra.run.smithi139.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 42 MB/s | 225 kB 00:00 2021-09-06T11:21:45.610 INFO:teuthology.orchestra.run.smithi139.stdout:(4/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2021-09-06T11:21:45.639 INFO:teuthology.orchestra.run.smithi139.stdout:(5/8): libgfortran-8.4.1-1.el8.x86_64.rpm 22 MB/s | 639 kB 00:00 2021-09-06T11:21:45.648 INFO:teuthology.orchestra.run.smithi139.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 18 MB/s | 169 kB 00:00 2021-09-06T11:21:45.783 INFO:teuthology.orchestra.run.smithi139.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.4.0+79 59 MB/s | 15 MB 00:00 2021-09-06T11:21:46.403 INFO:teuthology.orchestra.run.smithi139.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-747 7.2 MB/s | 7.4 MB 00:01 2021-09-06T11:21:46.406 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:21:46.406 INFO:teuthology.orchestra.run.smithi139.stdout:Total 35 MB/s | 36 MB 00:01 2021-09-06T11:21:46.415 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:21:46.433 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:21:46.434 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:21:46.793 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:21:46.796 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:21:47.365 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:21:47.508 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-06T11:21:47.650 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-06T11:21:47.841 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-06T11:21:48.373 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-06T11:21:48.574 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-0.3.12-1.el8.x86_64 3/8 2021-09-06T11:21:49.120 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 3/8 2021-09-06T11:21:49.257 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-06T11:21:49.976 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-06T11:21:50.331 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-09-06T11:21:50.632 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-06T11:21:52.619 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-06T11:21:53.845 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 7/8 2021-09-06T11:21:54.032 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7472.g2227f23 8/8 2021-09-06T11:21:54.339 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7472.g2227f23 8/8 2021-09-06T11:21:54.339 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7472.g2227f23 1/8 2021-09-06T11:21:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 2/8 2021-09-06T11:21:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-06T11:21:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-09-06T11:21:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-09-06T11:21:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 6/8 2021-09-06T11:21:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 7/8 2021-09-06T11:21:54.655 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 8/8 2021-09-06T11:21:54.655 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:54.655 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:21:54.655 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:21:54.656 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-09-06T11:21:54.656 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-09-06T11:21:54.656 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-0.3.12-1.el8.x86_64 2021-09-06T11:21:54.656 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-09-06T11:21:54.656 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-09-06T11:21:54.657 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-09-06T11:21:54.657 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x86_64 2021-09-06T11:21:54.657 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:54.657 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:21:54.720 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-rook 2021-09-06T11:21:55.162 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:35 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:21:56.121 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:21:56.123 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================== 2021-09-06T11:21:56.124 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-09-06T11:21:56.124 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================== 2021-09-06T11:21:56.124 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:21:56.124 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 44 k 2021-09-06T11:21:56.125 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-09-06T11:21:56.125 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-09-06T11:21:56.125 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-09-06T11:21:56.125 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-09-06T11:21:56.125 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-09-06T11:21:56.126 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-09-06T11:21:56.126 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-09-06T11:21:56.126 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-09-06T11:21:56.126 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-09-06T11:21:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-09-06T11:21:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-09-06T11:21:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-09-06T11:21:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-09-06T11:21:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:56.128 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:21:56.128 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================== 2021-09-06T11:21:56.128 INFO:teuthology.orchestra.run.smithi139.stdout:Install 13 Packages 2021-09-06T11:21:56.128 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:21:56.129 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 1.9 M 2021-09-06T11:21:56.129 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 25 M 2021-09-06T11:21:56.130 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:21:56.220 INFO:teuthology.orchestra.run.smithi139.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 283 kB/s | 27 kB 00:00 2021-09-06T11:21:56.230 INFO:teuthology.orchestra.run.smithi139.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 192 kB/s | 20 kB 00:00 2021-09-06T11:21:56.251 INFO:teuthology.orchestra.run.smithi139.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.0 MB/s | 126 kB 00:00 2021-09-06T11:21:56.259 INFO:teuthology.orchestra.run.smithi139.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 3.6 MB/s | 110 kB 00:00 2021-09-06T11:21:56.271 INFO:teuthology.orchestra.run.smithi139.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 3.5 MB/s | 43 kB 00:00 2021-09-06T11:21:56.272 INFO:teuthology.orchestra.run.smithi139.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 7.1 MB/s | 155 kB 00:00 2021-09-06T11:21:56.277 INFO:teuthology.orchestra.run.smithi139.stdout:(7/13): ceph-mgr-rook-17.0.0-7472.g2227f238.el8 290 kB/s | 44 kB 00:00 2021-09-06T11:21:56.346 INFO:teuthology.orchestra.run.smithi139.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 222 kB/s | 16 kB 00:00 2021-09-06T11:21:56.346 INFO:teuthology.orchestra.run.smithi139.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 443 kB/s | 33 kB 00:00 2021-09-06T11:21:56.375 INFO:teuthology.orchestra.run.smithi139.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 834 kB/s | 81 kB 00:00 2021-09-06T11:21:56.395 INFO:teuthology.orchestra.run.smithi139.stdout:(11/13): python3-rsa-3.4.2-16.el8.noarch.rpm 1.5 MB/s | 73 kB 00:00 2021-09-06T11:21:56.400 INFO:teuthology.orchestra.run.smithi139.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.4 MB/s | 61 kB 00:00 2021-09-06T11:21:56.518 INFO:teuthology.orchestra.run.smithi139.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.8 MB/s | 1.2 MB 00:00 2021-09-06T11:21:56.520 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:21:56.521 INFO:teuthology.orchestra.run.smithi139.stdout:Total 4.9 MB/s | 1.9 MB 00:00 2021-09-06T11:21:56.525 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:21:56.539 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:21:56.539 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:21:56.612 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:21:56.615 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:21:57.128 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:21:57.395 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-09-06T11:21:57.588 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-09-06T11:21:57.827 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-09-06T11:21:58.030 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-09-06T11:21:58.213 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-09-06T11:21:58.482 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-09-06T11:21:58.803 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-09-06T11:21:59.868 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-09-06T11:22:00.960 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-09-06T11:22:01.229 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-09-06T11:22:01.396 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-09-06T11:22:01.606 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-09-06T11:22:01.805 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-rook-2:17.0.0-7472.g2227f238.el8.noarch 13/13 2021-09-06T11:22:02.035 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7472.g2227f238.el8.noarch 13/13 2021-09-06T11:22:02.036 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7472.g2227f238.el8.noarch 1/13 2021-09-06T11:22:02.036 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-09-06T11:22:02.036 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-09-06T11:22:02.037 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-09-06T11:22:02.037 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-09-06T11:22:02.037 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-09-06T11:22:02.037 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-09-06T11:22:02.038 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-09-06T11:22:02.038 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-09-06T11:22:02.038 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-09-06T11:22:02.038 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-09-06T11:22:02.039 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-09-06T11:22:02.395 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-09-06T11:22:02.396 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:02.396 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:22:02.396 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:22:02.396 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-09-06T11:22:02.397 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-09-06T11:22:02.397 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-09-06T11:22:02.397 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-09-06T11:22:02.397 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-09-06T11:22:02.397 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-09-06T11:22:02.398 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-09-06T11:22:02.398 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-09-06T11:22:02.398 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-09-06T11:22:02.398 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-09-06T11:22:02.399 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-09-06T11:22:02.399 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-09-06T11:22:02.399 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:02.399 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:02.512 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-cephadm 2021-09-06T11:22:02.955 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:42 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:03.926 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:03.927 INFO:teuthology.orchestra.run.smithi139.stdout:=================================================================================================================== 2021-09-06T11:22:03.928 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-09-06T11:22:03.928 INFO:teuthology.orchestra.run.smithi139.stdout:=================================================================================================================== 2021-09-06T11:22:03.929 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:22:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7472.g2227f238.el8 ceph-noarch 91 k 2021-09-06T11:22:03.929 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-09-06T11:22:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh noarch 2.7.0-1.el8 copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 476 k 2021-09-06T11:22:03.929 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-09-06T11:22:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-09-06T11:22:03.930 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:03.930 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:22:03.931 INFO:teuthology.orchestra.run.smithi139.stdout:=================================================================================================================== 2021-09-06T11:22:03.931 INFO:teuthology.orchestra.run.smithi139.stdout:Install 4 Packages 2021-09-06T11:22:03.931 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:03.931 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 5.9 M 2021-09-06T11:22:03.932 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 25 M 2021-09-06T11:22:03.932 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:22:03.980 INFO:teuthology.orchestra.run.smithi139.stdout:(1/4): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 10 MB/s | 538 kB 00:00 2021-09-06T11:22:04.020 INFO:teuthology.orchestra.run.smithi139.stdout:(2/4): python3-babel-2.5.1-5.el8.noarch.rpm 52 MB/s | 4.8 MB 00:00 2021-09-06T11:22:04.060 INFO:teuthology.orchestra.run.smithi139.stdout:(3/4): ceph-mgr-cephadm-17.0.0-7472.g2227f238.e 695 kB/s | 91 kB 00:00 2021-09-06T11:22:05.877 INFO:teuthology.orchestra.run.smithi139.stdout:(4/4): python3-asyncssh-2.7.0-1.el8.noarch.rpm 251 kB/s | 476 kB 00:01 2021-09-06T11:22:05.882 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:22:05.882 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.0 MB/s | 5.9 MB 00:01 2021-09-06T11:22:05.886 INFO:teuthology.orchestra.run.smithi139.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ceph:python3-asyncssh-5dfbfa110d1e0dad/packages/python3-asyncssh-2.7.0-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 45bf1189: NOKEY 2021-09-06T11:22:06.681 INFO:teuthology.orchestra.run.smithi139.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 1.0 kB 00:00 2021-09-06T11:22:06.784 INFO:teuthology.orchestra.run.smithi139.stderr:Importing GPG key 0x45BF1189: 2021-09-06T11:22:06.784 INFO:teuthology.orchestra.run.smithi139.stderr: Userid : "ceph_python3-asyncssh (None) " 2021-09-06T11:22:06.785 INFO:teuthology.orchestra.run.smithi139.stderr: Fingerprint: FD53 FC11 9FA9 2C47 E35D 9C35 339B 8950 45BF 1189 2021-09-06T11:22:06.785 INFO:teuthology.orchestra.run.smithi139.stderr: From : https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/pubkey.gpg 2021-09-06T11:22:06.879 INFO:teuthology.orchestra.run.smithi139.stdout:Key imported successfully 2021-09-06T11:22:06.881 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:22:06.890 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:22:06.890 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:22:06.976 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:22:06.979 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:22:07.359 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:22:08.361 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-asyncssh-2.7.0-1.el8.noarch 1/4 2021-09-06T11:22:08.739 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 2/4 2021-09-06T11:22:09.046 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-06T11:22:09.229 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7472.g2227f238.el8.noarch 4/4 2021-09-06T11:22:09.464 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7472.g2227f238.el8.noarch 4/4 2021-09-06T11:22:09.464 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7472.g2227f238.el8.noarch 1/4 2021-09-06T11:22:09.464 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/4 2021-09-06T11:22:09.465 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-06T11:22:09.753 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-asyncssh-2.7.0-1.el8.noarch 4/4 2021-09-06T11:22:09.753 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:09.753 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:22:09.754 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm-2:17.0.0-7472.g2227f238.el8.noarch 2021-09-06T11:22:09.754 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh-2.7.0-1.el8.noarch 2021-09-06T11:22:09.754 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel-2.5.1-5.el8.noarch 2021-09-06T11:22:09.754 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-09-06T11:22:09.755 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:09.755 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:09.812 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-fuse 2021-09-06T11:22:10.257 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:50 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:11.216 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:11.218 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:11.218 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-09-06T11:22:11.219 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:11.219 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:22:11.219 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse x86_64 2:17.0.0-7472.g2227f238.el8 ceph 868 k 2021-09-06T11:22:11.220 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-09-06T11:22:11.220 INFO:teuthology.orchestra.run.smithi139.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-09-06T11:22:11.221 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:11.221 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:22:11.221 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:11.222 INFO:teuthology.orchestra.run.smithi139.stdout:Install 2 Packages 2021-09-06T11:22:11.222 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:11.223 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 950 k 2021-09-06T11:22:11.223 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 3.1 M 2021-09-06T11:22:11.224 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:22:11.250 INFO:teuthology.orchestra.run.smithi139.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2021-09-06T11:22:11.452 INFO:teuthology.orchestra.run.smithi139.stdout:(2/2): ceph-fuse-17.0.0-7472.g2227f238.el8.x86_ 3.6 MB/s | 868 kB 00:00 2021-09-06T11:22:11.453 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:22:11.453 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.9 MB/s | 950 kB 00:00 2021-09-06T11:22:11.455 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:22:11.476 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:22:11.476 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:22:11.553 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:22:11.556 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:22:11.768 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:22:12.075 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-09-06T11:22:12.320 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 2/2 2021-09-06T11:22:12.661 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 2/2 2021-09-06T11:22:12.662 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 1/2 2021-09-06T11:22:12.982 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-09-06T11:22:12.982 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:12.983 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:22:12.983 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-09-06T11:22:12.983 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:12.983 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:13.208 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados-devel 2021-09-06T11:22:13.663 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:53 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:14.599 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:14.600 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2021-09-06T11:22:14.601 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:14.601 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:22:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel x86_64 2:17.0.0-7472.g2227f238.el8 ceph 125 k 2021-09-06T11:22:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:14.602 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:22:14.602 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:14.602 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-09-06T11:22:14.603 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:14.603 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 125 k 2021-09-06T11:22:14.603 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 432 k 2021-09-06T11:22:14.603 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:22:14.755 INFO:teuthology.orchestra.run.smithi139.stdout:librados-devel-17.0.0-7472.g2227f238.el8.x86_64 811 kB/s | 125 kB 00:00 2021-09-06T11:22:14.756 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:22:14.756 INFO:teuthology.orchestra.run.smithi139.stdout:Total 804 kB/s | 125 kB 00:00 2021-09-06T11:22:14.756 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:22:14.762 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:22:14.762 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:22:14.811 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:22:14.814 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:22:14.981 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:22:15.168 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:15.411 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:15.767 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:15.768 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:15.768 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:22:15.768 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:22:15.768 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:15.769 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:15.958 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs2 2021-09-06T11:22:16.412 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:56 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:17.109 INFO:teuthology.orchestra.run.smithi139.stdout:Package libcephfs2-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:22:17.359 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:17.360 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-09-06T11:22:17.360 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:17.418 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs-devel 2021-09-06T11:22:17.863 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:57 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:18.814 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:18.815 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:18.816 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2021-09-06T11:22:18.816 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:18.816 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:22:18.816 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel x86_64 2:17.0.0-7472.g2227f238.el8 ceph 24 k 2021-09-06T11:22:18.817 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:18.817 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:22:18.817 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:18.817 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-09-06T11:22:18.817 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:18.818 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 24 k 2021-09-06T11:22:18.818 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 102 k 2021-09-06T11:22:18.818 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:22:18.940 INFO:teuthology.orchestra.run.smithi139.stdout:libcephfs-devel-17.0.0-7472.g2227f238.el8.x86_6 193 kB/s | 24 kB 00:00 2021-09-06T11:22:18.941 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:22:18.941 INFO:teuthology.orchestra.run.smithi139.stdout:Total 191 kB/s | 24 kB 00:00 2021-09-06T11:22:18.942 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:22:18.944 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:22:18.945 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:22:18.946 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:22:18.949 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:22:18.998 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:22:19.250 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:19.473 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:19.708 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs-devel-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:19.709 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:19.709 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:22:19.709 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:22:19.709 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:19.710 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:19.772 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados2 2021-09-06T11:22:20.217 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:00 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:21.377 INFO:teuthology.orchestra.run.smithi139.stdout:Package librados2-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:22:21.632 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:21.633 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-09-06T11:22:21.634 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:21.966 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librbd1 2021-09-06T11:22:22.419 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:02 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:23.141 INFO:teuthology.orchestra.run.smithi139.stdout:Package librbd1-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:22:23.392 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:23.393 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-09-06T11:22:23.394 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:23.448 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rados 2021-09-06T11:22:23.894 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:03 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:24.599 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rados-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:22:24.850 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:24.851 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-09-06T11:22:24.851 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:24.896 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rgw 2021-09-06T11:22:25.340 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:05 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:26.071 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rgw-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:22:26.331 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:26.332 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-09-06T11:22:26.332 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:26.379 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-cephfs 2021-09-06T11:22:26.824 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:06 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:27.741 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-cephfs-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:22:27.991 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:27.992 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-09-06T11:22:27.992 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:28.044 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rbd 2021-09-06T11:22:28.486 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:08 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:29.190 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rbd-2:17.0.0-7472.g2227f238.el8.x86_64 is already installed. 2021-09-06T11:22:29.440 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:29.441 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-09-06T11:22:29.442 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:29.486 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-fuse 2021-09-06T11:22:29.930 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:09 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:30.897 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:30.898 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:30.899 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2021-09-06T11:22:30.899 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:30.899 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:22:30.899 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse x86_64 2:17.0.0-7472.g2227f238.el8 ceph 93 k 2021-09-06T11:22:30.900 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:30.900 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:22:30.900 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:30.900 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-09-06T11:22:30.901 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:30.901 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 93 k 2021-09-06T11:22:30.901 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 281 k 2021-09-06T11:22:30.901 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:22:31.031 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-fuse-17.0.0-7472.g2227f238.el8.x86_64.rpm 706 kB/s | 93 kB 00:00 2021-09-06T11:22:31.032 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:22:31.032 INFO:teuthology.orchestra.run.smithi139.stdout:Total 699 kB/s | 93 kB 00:00 2021-09-06T11:22:31.032 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:22:31.043 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:22:31.043 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:22:31.093 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:22:31.096 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:22:31.259 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:22:31.457 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:31.697 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:31.981 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:31.982 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:31.982 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:22:31.982 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:22:31.983 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:31.983 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:32.177 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-mirror 2021-09-06T11:22:32.630 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:12 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:33.665 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:33.666 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:33.666 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2021-09-06T11:22:33.667 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:33.667 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:22:33.667 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror x86_64 2:17.0.0-7472.g2227f238.el8 ceph 3.8 M 2021-09-06T11:22:33.667 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:33.668 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:22:33.668 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:33.668 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-09-06T11:22:33.668 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:33.669 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 3.8 M 2021-09-06T11:22:33.669 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 19 M 2021-09-06T11:22:33.669 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:22:33.991 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-mirror-17.0.0-7472.g2227f238.el8.x86_64.rpm 12 MB/s | 3.8 MB 00:00 2021-09-06T11:22:33.991 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:22:33.992 INFO:teuthology.orchestra.run.smithi139.stdout:Total 12 MB/s | 3.8 MB 00:00 2021-09-06T11:22:33.992 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:22:34.005 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:22:34.005 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:22:34.068 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:22:34.072 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:22:34.544 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:22:34.708 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:35.075 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:35.407 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:35.407 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:35.407 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:22:35.408 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:22:35.408 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:35.408 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:35.537 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-nbd 2021-09-06T11:22:35.990 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:15 ago on Mon 06 Sep 2021 11:20:20 AM UTC. 2021-09-06T11:22:36.936 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-09-06T11:22:36.937 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:36.937 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2021-09-06T11:22:36.937 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:36.938 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-09-06T11:22:36.938 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd x86_64 2:17.0.0-7472.g2227f238.el8 ceph 173 k 2021-09-06T11:22:36.938 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:36.938 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-09-06T11:22:36.939 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-09-06T11:22:36.939 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-09-06T11:22:36.939 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:36.939 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 173 k 2021-09-06T11:22:36.939 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 563 k 2021-09-06T11:22:36.940 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-09-06T11:22:37.091 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-nbd-17.0.0-7472.g2227f238.el8.x86_64.rpm 1.1 MB/s | 173 kB 00:00 2021-09-06T11:22:37.092 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-09-06T11:22:37.092 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.1 MB/s | 173 kB 00:00 2021-09-06T11:22:37.092 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-09-06T11:22:37.104 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-09-06T11:22:37.104 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-09-06T11:22:37.153 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-09-06T11:22:37.156 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-09-06T11:22:37.298 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-09-06T11:22:37.481 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:37.712 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:37.989 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-2:17.0.0-7472.g2227f238.el8.x86_64 1/1 2021-09-06T11:22:37.990 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:37.990 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-09-06T11:22:37.990 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-2:17.0.0-7472.g2227f238.el8.x86_64 2021-09-06T11:22:37.990 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:37.991 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-09-06T11:22:38.184 DEBUG:teuthology.parallel:result is None 2021-09-06T11:22:38.185 DEBUG:teuthology.orchestra.run.smithi094:> sudo lsb_release -is 2021-09-06T11:22:38.239 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS 2021-09-06T11:22:38.240 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-06T11:22:38.241 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2227f2382f1df5117a62687acd2d99ea5eeb871e 2021-09-06T11:22:38.395 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-09-06T11:22:38.416 INFO:teuthology.orchestra.run.smithi094.stdout:17.0.0-7472.g2227f238.el8 2021-09-06T11:22:38.417 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7472.g2227f238.el8 2021-09-06T11:22:38.418 INFO:teuthology.task.install:The correct ceph version 17.0.0-7472.g2227f238 is installed. 2021-09-06T11:22:38.418 DEBUG:teuthology.orchestra.run.smithi139:> sudo lsb_release -is 2021-09-06T11:22:38.451 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS 2021-09-06T11:22:38.452 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-06T11:22:38.453 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2227f2382f1df5117a62687acd2d99ea5eeb871e 2021-09-06T11:22:38.612 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-09-06T11:22:38.631 INFO:teuthology.orchestra.run.smithi139.stdout:17.0.0-7472.g2227f238.el8 2021-09-06T11:22:38.631 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7472.g2227f238.el8 2021-09-06T11:22:38.632 INFO:teuthology.task.install:The correct ceph version 17.0.0-7472.g2227f238 is installed. 2021-09-06T11:22:38.632 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-09-06T11:22:38.633 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:38.633 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-06T11:22:38.662 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:22:38.662 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-06T11:22:38.695 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-09-06T11:22:38.695 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:38.695 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2021-09-06T11:22:38.728 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-06T11:22:38.793 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:22:38.794 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2021-09-06T11:22:38.817 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-06T11:22:38.881 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-09-06T11:22:38.882 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:38.882 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-06T11:22:38.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-06T11:22:38.973 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:22:38.973 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-06T11:22:38.996 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-06T11:22:39.059 INFO:teuthology.run_tasks:Running task ceph... 2021-09-06T11:22:39.122 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-09-06T11:22:39.123 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2021-09-06T11:22:39.126 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /var/log/ceph 2021-09-06T11:22:39.149 INFO:tasks.ceph:Disabling ceph logrotate... 2021-09-06T11:22:39.150 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-06T11:22:39.191 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-06T11:22:39.214 INFO:tasks.ceph:Creating extra log directories... 2021-09-06T11:22:39.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-06T11:22:39.256 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-06T11:22:39.290 INFO:tasks.ceph:Creating ceph cluster c1... 2021-09-06T11:22:39.290 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-06T11:22:39.291 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi/6376787', 'branch': 'wip-skoduri-dbstore-vstart', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6376787', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.3', 'overrides': {'admin_socket': {'branch': 'wip-skoduri-dbstore-vstart'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-skoduri-dbstore-vstart', 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e'}}, 'owner': 'scheduled_soumyakoduri@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-skoduri-dbstore-vstart', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_2227f2382f1df5117a62687acd2d99ea5eeb871e/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'targets': {'smithi094.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUTN53Z3n5QlqTDgk0NpEVg01dFbGUi6kCFNTkBAjTvi+FP0HPPz4m/nWJJWMgn9J9x87EXF9d5j4yhDZZUbFiKnCtE0dFqp7yDt+646ytY2pcOWqdnv3fMuK2YhdkRApWI6PATZkxRTQwnfDUxTWfHFjtREnNxDmZfc2W0YrsU3gCqByrMTHrycORc7y3OUyenla28nMmmYOIRAOJmwJi5dZBtxSK85jeYMra3DXk/yW3puzjQiyWMtUhdo4xEE/dnf9rAENBmVX7dCyo1Kei73l5gzG0b5dlipQcx3coHbU+P6YpHmRbxYRNFGX2LKiW2Qe9N0tS4EYmfK2kRsvcurg9UYzOof3BHPHYDAFygcudl9WTELEmtRTOYXltluuzCFxpnO6S8z+pfnExOIx9aPCm0dvzJlQyfOTFs+OggDmYQOPlQnoiZLEbfukHyPaig4GpWwscWbmWsOPTqwF2PPyOoATGx5+xfcHkmYmM26ahbjySb/iMipmGwvdWwZU=', 'smithi139.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxOSAXu0ahF2fafU0XYIhsUwAZYZNMc0rU5nwzbFpNLJIohFw7h/mcnGZ+vhFkODxC9wceI+y+q0Ku77hekOeNq2jhxMbe9ursZc16VNyEGSg8FYaJQ/pxeHtRGF7kpKP6Tjq1qTHnRIUu+8TuWjnw7F85vyH64WvQORUNGmeH3OEz3lv+ZQGFLO5v2w1Vux/q42+sYA6KGNSFqXGfYDha3W3HUmS+d+K/z6ZIz1ZUGN4eCikpM2wL9gcirFUOKnCLxjgrDI9cTZY01z21PQG4HeYoaH7ZTFO2rWVkzAZLPhhHVRNzgLJe1fIuAcFG3GOplFJI4GEWg+VOQNsh4+SG1Mm1rrWZSYrwQ74bNDrFD8kCnLfNcKqpo0t2Qh1IqbPOAcrWoAqnoMFzOlh1kkc0WsF30uEhQVnHkIlF0yZfWhKuHx0gwG9+jotBkxUImCL1Pj1hzupFeFIMdhVY7PLurYFJtwqvKFXl7WeWoub2qHGq7S6tbv6oqxQlksp2M/E='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-09-06_10:28:39', 'tube': 'smithi', 'user': 'soumyakoduri', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-06T11:22:39.292 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-06T11:22:39.320 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-06T11:22:39.334 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2021-09-06T11:22:39.375 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/run/ceph 2021-09-06T11:22:39.398 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:39.399 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2021-09-06T11:22:39.451 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-06T11:22:39.451 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-06T11:22:39.504 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-06T11:22:39.505 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T11:22:39.505 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 138429 Links: 1 2021-09-06T11:22:39.505 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T11:22:39.505 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T11:22:39.506 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2021-09-06 11:13:59.103706452 +0000 2021-09-06T11:22:39.506 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2021-09-06 11:03:52.961751584 +0000 2021-09-06T11:22:39.506 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2021-09-06 11:03:52.961751584 +0000 2021-09-06T11:22:39.506 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2021-09-06T11:22:39.514 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2021-09-06T11:22:39.514 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2021-09-06T11:22:39.514 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 8.1179e-05 s, 6.3 MB/s 2021-09-06T11:22:39.516 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-06T11:22:39.571 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-06T11:22:39.571 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T11:22:39.571 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 138455 Links: 1 2021-09-06T11:22:39.571 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T11:22:39.572 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T11:22:39.572 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2021-09-06 11:13:59.103706452 +0000 2021-09-06T11:22:39.572 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2021-09-06 11:03:53.287746114 +0000 2021-09-06T11:22:39.572 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2021-09-06 11:03:53.287746114 +0000 2021-09-06T11:22:39.572 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2021-09-06T11:22:39.580 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2021-09-06T11:22:39.580 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2021-09-06T11:22:39.580 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 9.5773e-05 s, 5.3 MB/s 2021-09-06T11:22:39.583 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-06T11:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-06T11:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T11:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 141420 Links: 1 2021-09-06T11:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T11:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T11:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2021-09-06 11:13:59.103706452 +0000 2021-09-06T11:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2021-09-06 11:03:53.776737909 +0000 2021-09-06T11:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2021-09-06 11:03:53.776737909 +0000 2021-09-06T11:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2021-09-06T11:22:39.645 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2021-09-06T11:22:39.646 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2021-09-06T11:22:39.646 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000104234 s, 4.9 MB/s 2021-09-06T11:22:39.648 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-06T11:22:39.701 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-06T11:22:39.702 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T11:22:39.702 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 142395 Links: 1 2021-09-06T11:22:39.702 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T11:22:39.702 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T11:22:39.703 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2021-09-06 11:13:59.103706452 +0000 2021-09-06T11:22:39.703 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2021-09-06 11:03:54.102732440 +0000 2021-09-06T11:22:39.703 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2021-09-06 11:03:54.102732440 +0000 2021-09-06T11:22:39.703 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2021-09-06T11:22:39.710 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2021-09-06T11:22:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2021-09-06T11:22:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000112561 s, 4.5 MB/s 2021-09-06T11:22:39.713 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T11:22:39.713 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-06T11:22:39.714 INFO:tasks.ceph:Generating config... 2021-09-06T11:22:39.715 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-06T11:22:39.715 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-06T11:22:39.716 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-06T11:22:39.716 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T11:22:39.716 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-06T11:22:39.716 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-06T11:22:39.717 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-06T11:22:39.717 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-06T11:22:39.717 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-06T11:22:39.718 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-06T11:22:39.718 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-06T11:22:39.718 INFO:tasks.ceph:[client] setuser = ceph 2021-09-06T11:22:39.719 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-06T11:22:39.719 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-06T11:22:39.719 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-06T11:22:39.719 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-06T11:22:39.720 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-06T11:22:39.720 INFO:tasks.ceph:[osd] debug filestore = 20 2021-09-06T11:22:39.720 INFO:tasks.ceph:[osd] debug journal = 20 2021-09-06T11:22:39.721 INFO:tasks.ceph:[osd] debug ms = 20 2021-09-06T11:22:39.721 INFO:tasks.ceph:[osd] debug osd = 25 2021-09-06T11:22:39.721 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-06T11:22:39.722 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-06T11:22:39.722 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-06T11:22:39.722 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-06T11:22:39.722 INFO:tasks.ceph:Setting up c1.mon.a... 2021-09-06T11:22:39.723 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-09-06T11:22:39.809 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/c1.keyring 2021-09-06T11:22:39.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-09-06T11:22:39.848 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-06T11:22:39.911 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.94')] 2021-09-06T11:22:39.912 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.94'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-06T11:22:39.913 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:39.913 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-06T11:22:39.966 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.94 --print /home/ubuntu/cephtest/c1.monmap 2021-09-06T11:22:40.039 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:22:40.039 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:22:40.040 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-09-06T11:22:40.041 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: generated fsid dd8e0d3d-b2da-445f-9f1a-101a0b6989fd 2021-09-06T11:22:40.041 INFO:teuthology.orchestra.run.smithi094.stdout:setting min_mon_release = octopus 2021-09-06T11:22:40.041 INFO:teuthology.orchestra.run.smithi094.stdout:epoch 0 2021-09-06T11:22:40.041 INFO:teuthology.orchestra.run.smithi094.stdout:fsid dd8e0d3d-b2da-445f-9f1a-101a0b6989fd 2021-09-06T11:22:40.042 INFO:teuthology.orchestra.run.smithi094.stdout:last_changed 2021-09-06T11:22:40.040920+0000 2021-09-06T11:22:40.042 INFO:teuthology.orchestra.run.smithi094.stdout:created 2021-09-06T11:22:40.040920+0000 2021-09-06T11:22:40.042 INFO:teuthology.orchestra.run.smithi094.stdout:min_mon_release 15 (octopus) 2021-09-06T11:22:40.042 INFO:teuthology.orchestra.run.smithi094.stdout:election_strategy: 1 2021-09-06T11:22:40.043 INFO:teuthology.orchestra.run.smithi094.stdout:0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2021-09-06T11:22:40.043 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-09-06T11:22:40.044 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-06T11:22:40.095 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID dd8e0d3d-b2da-445f-9f1a-101a0b6989fd... 2021-09-06T11:22:40.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-06T11:22:40.137 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-06T11:22:40.175 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2021-09-06T11:22:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2021-09-06T11:22:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-06T11:22:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2021-09-06T11:22:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2021-09-06T11:22:40.176 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2021-09-06T11:22:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2021-09-06T11:22:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2021-09-06T11:22:40.177 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2021-09-06T11:22:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2021-09-06T11:22:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.178 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-06T11:22:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2021-09-06T11:22:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2021-09-06T11:22:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-06T11:22:40.179 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2021-09-06T11:22:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2021-09-06T11:22:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2021-09-06T11:22:40.180 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2021-09-06T11:22:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2021-09-06T11:22:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2021-09-06T11:22:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2021-09-06T11:22:40.181 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2021-09-06T11:22:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-06T11:22:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-06T11:22:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2021-09-06T11:22:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.182 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-06T11:22:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2021-09-06T11:22:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2021-09-06T11:22:40.183 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2021-09-06T11:22:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2021-09-06T11:22:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2021-09-06T11:22:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = 172.21.15.94 2021-09-06T11:22:40.184 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = dd8e0d3d-b2da-445f-9f1a-101a0b6989fd 2021-09-06T11:22:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.185 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2021-09-06T11:22:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2021-09-06T11:22:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.185 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2021-09-06T11:22:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2021-09-06T11:22:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2021-09-06T11:22:40.186 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2021-09-06T11:22:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2021-09-06T11:22:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2021-09-06T11:22:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2021-09-06T11:22:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.187 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2021-09-06T11:22:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2021-09-06T11:22:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2021-09-06T11:22:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.188 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2021-09-06T11:22:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2021-09-06T11:22:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2021-09-06T11:22:40.189 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2021-09-06T11:22:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2021-09-06T11:22:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2021-09-06T11:22:40.190 INFO:teuthology.orchestra.run.smithi094.stdout: debug filestore = 20 2021-09-06T11:22:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: debug journal = 20 2021-09-06T11:22:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 20 2021-09-06T11:22:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 25 2021-09-06T11:22:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: rocksdb delete range threshold = 0 2021-09-06T11:22:40.191 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.191 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2021-09-06T11:22:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2021-09-06T11:22:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2021-09-06T11:22:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2021-09-06T11:22:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2021-09-06T11:22:40.192 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2021-09-06T11:22:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2021-09-06T11:22:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2021-09-06T11:22:40.193 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.194 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2021-09-06T11:22:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2021-09-06T11:22:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2021-09-06T11:22:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2021-09-06T11:22:40.194 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2021-09-06T11:22:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2021-09-06T11:22:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2021-09-06T11:22:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2021-09-06T11:22:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2021-09-06T11:22:40.195 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2021-09-06T11:22:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2021-09-06T11:22:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-06T11:22:40.196 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2021-09-06T11:22:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2021-09-06T11:22:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2021-09-06T11:22:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-06T11:22:40.197 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-06T11:22:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:40.198 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2021-09-06T11:22:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2021-09-06T11:22:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2021-09-06T11:22:40.198 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2021-09-06T11:22:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-06T11:22:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-06T11:22:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: debug rgw = 20 2021-09-06T11:22:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt require ssl = False 2021-09-06T11:22:40.199 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt s3 kms backend = testing 2021-09-06T11:22:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T11:22:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: rgw curl low speed time = 300 2021-09-06T11:22:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: rgw data log num shards = 4 2021-09-06T11:22:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: rgw md log max shards = 4 2021-09-06T11:22:40.200 INFO:teuthology.orchestra.run.smithi094.stdout: rgw sync log trim interval = 0 2021-09-06T11:22:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: rgw sync meta inject err probability = 0.1 2021-09-06T11:22:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: rgw sync obj etag verify = True 2021-09-06T11:22:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: setgroup = ceph 2021-09-06T11:22:40.201 INFO:teuthology.orchestra.run.smithi094.stdout: setuser = ceph 2021-09-06T11:22:40.202 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2021-09-06T11:22:40.202 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2021-09-06T11:22:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 10 2021-09-06T11:22:40.202 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2021-09-06T11:22:40.202 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 1000 2021-09-06T11:22:40.203 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2021-09-06T11:22:40.203 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 10000 2021-09-06T11:22:40.204 INFO:teuthology.orchestra.run.smithi139.stdout:[global] 2021-09-06T11:22:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: chdir = "" 2021-09-06T11:22:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-06T11:22:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: auth supported = cephx 2021-09-06T11:22:40.205 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: filestore xattr use omap = true 2021-09-06T11:22:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: mon clock drift allowed = 1.000 2021-09-06T11:22:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.206 INFO:teuthology.orchestra.run.smithi139.stdout: osd crush chooseleaf type = 0 2021-09-06T11:22:40.207 INFO:teuthology.orchestra.run.smithi139.stdout: auth debug = true 2021-09-06T11:22:40.207 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.207 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on old message = true 2021-09-06T11:22:40.207 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on bug = true 2021-09-06T11:22:40.208 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.208 INFO:teuthology.orchestra.run.smithi139.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-06T11:22:40.208 INFO:teuthology.orchestra.run.smithi139.stdout: mon pg warn max object skew = 0 2021-09-06T11:22:40.208 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.209 INFO:teuthology.orchestra.run.smithi139.stdout: # disable pg_autoscaler by default for new pools 2021-09-06T11:22:40.209 INFO:teuthology.orchestra.run.smithi139.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-06T11:22:40.209 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.209 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default size = 2 2021-09-06T11:22:40.209 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.210 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow primary affinity = true 2021-09-06T11:22:40.210 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow pg remap = true 2021-09-06T11:22:40.210 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on legacy crush tunables = false 2021-09-06T11:22:40.210 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on crush straw calc version zero = false 2021-09-06T11:22:40.210 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on no sortbitwise = false 2021-09-06T11:22:40.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on osd down out interval zero = false 2021-09-06T11:22:40.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on too few osds = false 2021-09-06T11:22:40.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-06T11:22:40.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-06T11:22:40.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon_allow_pool_size_one = true 2021-09-06T11:22:40.212 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.212 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-06T11:22:40.212 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.212 INFO:teuthology.orchestra.run.smithi139.stdout: osd default data pool replay window = 5 2021-09-06T11:22:40.212 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.213 INFO:teuthology.orchestra.run.smithi139.stdout: mon allow pool delete = true 2021-09-06T11:22:40.213 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.213 INFO:teuthology.orchestra.run.smithi139.stdout: mon cluster log file level = debug 2021-09-06T11:22:40.213 INFO:teuthology.orchestra.run.smithi139.stdout: debug asserts on shutdown = true 2021-09-06T11:22:40.213 INFO:teuthology.orchestra.run.smithi139.stdout: mon health detail to clog = false 2021-09-06T11:22:40.214 INFO:teuthology.orchestra.run.smithi139.stdout: mon host = 172.21.15.94 2021-09-06T11:22:40.214 INFO:teuthology.orchestra.run.smithi139.stdout: fsid = dd8e0d3d-b2da-445f-9f1a-101a0b6989fd 2021-09-06T11:22:40.214 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.214 INFO:teuthology.orchestra.run.smithi139.stdout:[osd] 2021-09-06T11:22:40.214 INFO:teuthology.orchestra.run.smithi139.stdout: osd journal size = 100 2021-09-06T11:22:40.215 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.215 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub load threshold = 5.0 2021-09-06T11:22:40.215 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub max interval = 600 2021-09-06T11:22:40.215 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.215 INFO:teuthology.orchestra.run.smithi139.stdout: osd recover clone overlap = true 2021-09-06T11:22:40.216 INFO:teuthology.orchestra.run.smithi139.stdout: osd recovery max chunk = 1048576 2021-09-06T11:22:40.216 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.216 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug shutdown = true 2021-09-06T11:22:40.216 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug op order = true 2021-09-06T11:22:40.217 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify stray on activate = true 2021-09-06T11:22:40.217 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.217 INFO:teuthology.orchestra.run.smithi139.stdout: osd open classes on start = true 2021-09-06T11:22:40.217 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug pg log writeout = true 2021-09-06T11:22:40.217 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.218 INFO:teuthology.orchestra.run.smithi139.stdout: osd deep scrub update digest min age = 30 2021-09-06T11:22:40.218 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.218 INFO:teuthology.orchestra.run.smithi139.stdout: osd map max advance = 10 2021-09-06T11:22:40.218 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.218 INFO:teuthology.orchestra.run.smithi139.stdout: journal zero on create = true 2021-09-06T11:22:40.219 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.219 INFO:teuthology.orchestra.run.smithi139.stdout: filestore ondisk finisher threads = 3 2021-09-06T11:22:40.219 INFO:teuthology.orchestra.run.smithi139.stdout: filestore apply finisher threads = 3 2021-09-06T11:22:40.219 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.219 INFO:teuthology.orchestra.run.smithi139.stdout: bdev debug aio = true 2021-09-06T11:22:40.219 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug misdirected ops = true 2021-09-06T11:22:40.220 INFO:teuthology.orchestra.run.smithi139.stdout: debug filestore = 20 2021-09-06T11:22:40.220 INFO:teuthology.orchestra.run.smithi139.stdout: debug journal = 20 2021-09-06T11:22:40.220 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 20 2021-09-06T11:22:40.220 INFO:teuthology.orchestra.run.smithi139.stdout: debug osd = 25 2021-09-06T11:22:40.220 INFO:teuthology.orchestra.run.smithi139.stdout: rocksdb delete range threshold = 0 2021-09-06T11:22:40.221 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.221 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr] 2021-09-06T11:22:40.221 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2021-09-06T11:22:40.221 INFO:teuthology.orchestra.run.smithi139.stdout: debug mgr = 20 2021-09-06T11:22:40.221 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2021-09-06T11:22:40.222 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2021-09-06T11:22:40.222 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2021-09-06T11:22:40.222 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2021-09-06T11:22:40.222 INFO:teuthology.orchestra.run.smithi139.stdout: mgr/telemetry/nag = false 2021-09-06T11:22:40.222 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.223 INFO:teuthology.orchestra.run.smithi139.stdout:[mon] 2021-09-06T11:22:40.223 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2021-09-06T11:22:40.223 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2021-09-06T11:22:40.223 INFO:teuthology.orchestra.run.smithi139.stdout: debug paxos = 20 2021-09-06T11:22:40.223 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2021-09-06T11:22:40.224 INFO:teuthology.orchestra.run.smithi139.stdout: mon data avail warn = 5 2021-09-06T11:22:40.224 INFO:teuthology.orchestra.run.smithi139.stdout: mon mgr mkfs grace = 240 2021-09-06T11:22:40.224 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2021-09-06T11:22:40.224 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd reporter subtree level = osd 2021-09-06T11:22:40.224 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd prime pg temp = true 2021-09-06T11:22:40.225 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2021-09-06T11:22:40.225 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.225 INFO:teuthology.orchestra.run.smithi139.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-06T11:22:40.225 INFO:teuthology.orchestra.run.smithi139.stdout: auth mon ticket ttl = 660 # 11m 2021-09-06T11:22:40.225 INFO:teuthology.orchestra.run.smithi139.stdout: auth service ticket ttl = 240 # 4m 2021-09-06T11:22:40.226 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.226 INFO:teuthology.orchestra.run.smithi139.stdout: # don't complain about insecure global_id in the test suite 2021-09-06T11:22:40.226 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-06T11:22:40.226 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-06T11:22:40.226 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:22:40.227 INFO:teuthology.orchestra.run.smithi139.stdout:[client] 2021-09-06T11:22:40.227 INFO:teuthology.orchestra.run.smithi139.stdout: rgw cache enabled = true 2021-09-06T11:22:40.227 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable ops log = true 2021-09-06T11:22:40.227 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable usage log = true 2021-09-06T11:22:40.227 INFO:teuthology.orchestra.run.smithi139.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-06T11:22:40.228 INFO:teuthology.orchestra.run.smithi139.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-06T11:22:40.228 INFO:teuthology.orchestra.run.smithi139.stdout: debug rgw = 20 2021-09-06T11:22:40.228 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt require ssl = False 2021-09-06T11:22:40.228 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt s3 kms backend = testing 2021-09-06T11:22:40.228 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T11:22:40.229 INFO:teuthology.orchestra.run.smithi139.stdout: rgw curl low speed time = 300 2021-09-06T11:22:40.229 INFO:teuthology.orchestra.run.smithi139.stdout: rgw data log num shards = 4 2021-09-06T11:22:40.229 INFO:teuthology.orchestra.run.smithi139.stdout: rgw md log max shards = 4 2021-09-06T11:22:40.229 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync log trim interval = 0 2021-09-06T11:22:40.229 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync meta inject err probability = 0.1 2021-09-06T11:22:40.230 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync obj etag verify = True 2021-09-06T11:22:40.230 INFO:teuthology.orchestra.run.smithi139.stdout: setgroup = ceph 2021-09-06T11:22:40.230 INFO:teuthology.orchestra.run.smithi139.stdout: setuser = ceph 2021-09-06T11:22:40.230 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.a] 2021-09-06T11:22:40.230 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.0] 2021-09-06T11:22:40.231 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 10 2021-09-06T11:22:40.231 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.1] 2021-09-06T11:22:40.231 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 1000 2021-09-06T11:22:40.231 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.2] 2021-09-06T11:22:40.231 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 10000 2021-09-06T11:22:40.232 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-09-06T11:22:40.232 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-09-06T11:22:40.293 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-06T11:22:40.293 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:40.294 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-09-06T11:22:40.308 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:40.308 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-09-06T11:22:40.361 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2021-09-06T11:22:40.362 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:40.362 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/c1.keyring 2021-09-06T11:22:40.363 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-06T11:22:40.433 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:40.434 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-06T11:22:40.488 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi139.front.sepia.ceph.com 2021-09-06T11:22:40.489 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:22:40.489 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/c1.keyring 2021-09-06T11:22:40.489 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-06T11:22:40.528 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:22:40.528 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-06T11:22:40.584 INFO:tasks.ceph:Setting up mon nodes... 2021-09-06T11:22:40.585 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-06T11:22:40.585 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-09-06T11:22:40.631 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-09-06T11:22:40.633 INFO:tasks.ceph:Setting up mds nodes... 2021-09-06T11:22:40.633 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-06T11:22:40.634 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-09-06T11:22:40.670 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/c1.client.0.keyring 2021-09-06T11:22:40.682 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-09-06T11:22:40.757 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/c1.client.1.keyring 2021-09-06T11:22:40.770 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-06T11:22:40.770 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-06T11:22:40.770 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-09-06T11:22:40.832 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T11:22:40.833 INFO:tasks.ceph:role: c1.osd.0 2021-09-06T11:22:40.833 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2021-09-06T11:22:40.833 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-06T11:22:40.895 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T11:22:40.895 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T11:22:40.896 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T11:22:40.896 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 2021-09-06T11:22:40.896 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T11:22:40.896 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2021-09-06T11:22:40.897 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T11:22:40.897 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T11:22:40.897 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T11:22:40.897 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T11:22:40.918 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2021-09-06T11:22:40.919 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2021-09-06T11:22:40.920 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-09-06T11:22:40.996 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-09-06T11:22:41.064 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-09-06T11:22:41.128 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T11:22:41.129 INFO:tasks.ceph:role: c1.osd.1 2021-09-06T11:22:41.129 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com 2021-09-06T11:22:41.129 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-06T11:22:41.191 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T11:22:41.191 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T11:22:41.192 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T11:22:41.192 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 2021-09-06T11:22:41.192 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T11:22:41.192 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2021-09-06T11:22:41.193 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T11:22:41.193 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T11:22:41.193 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T11:22:41.193 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T11:22:41.214 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2021-09-06T11:22:41.216 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2021-09-06T11:22:41.216 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-09-06T11:22:41.291 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-09-06T11:22:41.358 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-09-06T11:22:41.422 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T11:22:41.422 INFO:tasks.ceph:role: c1.osd.2 2021-09-06T11:22:41.423 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com 2021-09-06T11:22:41.423 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-06T11:22:41.485 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T11:22:41.485 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T11:22:41.485 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T11:22:41.485 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 2021-09-06T11:22:41.486 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T11:22:41.486 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2021-09-06T11:22:41.486 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T11:22:41.486 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T11:22:41.487 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T11:22:41.487 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T11:22:41.508 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2021-09-06T11:22:41.509 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2021-09-06T11:22:41.510 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-09-06T11:22:41.585 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-09-06T11:22:41.652 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-06T11:22:41.739 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:41.738+0000 7f03fcea60c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-09-06T11:22:41.739 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:41.738+0000 7f03fcea60c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-09-06T11:22:41.740 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:41.738+0000 7f03fcea60c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-06T11:22:41.740 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:41.738+0000 7f03fcea60c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-06T11:22:41.740 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:41.739+0000 7f03fcea60c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-06T11:22:41.740 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:41.739+0000 7f03fcea60c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-09-06T11:22:43.816 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-09-06T11:22:43.845 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-06T11:22:43.928 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:43.928+0000 7f8800e900c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-09-06T11:22:43.929 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:43.928+0000 7f8800e900c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-09-06T11:22:43.929 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:43.928+0000 7f8800e900c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-06T11:22:43.929 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:43.928+0000 7f8800e900c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-06T11:22:43.929 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:43.928+0000 7f8800e900c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-06T11:22:43.929 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:43.928+0000 7f8800e900c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-09-06T11:22:46.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-09-06T11:22:46.074 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-06T11:22:46.158 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:46.157+0000 7fc596e0f0c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-09-06T11:22:46.158 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:46.157+0000 7fc596e0f0c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-09-06T11:22:46.158 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:46.157+0000 7fc596e0f0c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-06T11:22:46.158 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:46.157+0000 7fc596e0f0c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-06T11:22:46.159 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:46.157+0000 7fc596e0f0c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-06T11:22:46.159 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:22:46.158+0000 7fc596e0f0c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-09-06T11:22:48.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-09-06T11:22:48.310 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-06T11:22:48.310 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:48.311 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-09-06T11:22:48.376 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:48.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-09-06T11:22:48.441 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:48.441 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-09-06T11:22:48.506 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:48.506 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-09-06T11:22:48.571 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:48.571 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-09-06T11:22:48.625 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:48.626 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2021-09-06T11:22:48.680 INFO:tasks.ceph:Adding keys to all mons... 2021-09-06T11:22:48.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo tee -a /etc/ceph/c1.keyring 2021-09-06T11:22:48.742 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.x] 2021-09-06T11:22:48.742 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAA+jVhdbWtJRAARrNvGa1aSMHjm9WavbwhCQ== 2021-09-06T11:22:48.743 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2021-09-06T11:22:48.743 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAB+jVhxGUaLBAAh72VKdyXyrtYLa608kyIlg== 2021-09-06T11:22:48.743 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2021-09-06T11:22:48.743 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAD+jVhAmpnNxAApqfPcM9wl6bICn+UICmYng== 2021-09-06T11:22:48.744 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2021-09-06T11:22:48.744 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAG+jVhUZR1CRAAMq4aiip/zlkKofZYZ543Iw== 2021-09-06T11:22:48.744 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2021-09-06T11:22:48.744 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAA+jVhtVwJKBAAFaNRRLvaj/Ou/oxoKz6ksQ== 2021-09-06T11:22:48.744 INFO:teuthology.orchestra.run.smithi094.stdout:[client.1] 2021-09-06T11:22:48.745 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAA+jVhrOs+LRAA5dlCWPl4IEBblbF2MnCMtw== 2021-09-06T11:22:48.746 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-06T11:22:48.831 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T11:22:48.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T11:22:48.961 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T11:22:49.005 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-06T11:22:49.054 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-06T11:22:49.102 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-06T11:22:49.102 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-09-06T11:22:49.125 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-09-06T11:22:49.215 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-09-06T11:22:49.239 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-09-06T11:22:49.292 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-09-06T11:22:49.293 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-09-06T11:22:49.294 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-09-06T11:22:49.335 INFO:tasks.ceph.c1.mon.a:Started 2021-09-06T11:22:49.335 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-09-06T11:22:49.336 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-09-06T11:22:49.336 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-09-06T11:22:49.338 INFO:tasks.ceph.c1.mgr.x:Started 2021-09-06T11:22:49.338 INFO:tasks.ceph:Setting crush tunables to default 2021-09-06T11:22:49.339 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd crush tunables default 2021-09-06T11:22:49.534 INFO:teuthology.orchestra.run.smithi094.stderr:adjusted tunables profile to default 2021-09-06T11:22:49.545 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-09-06T11:22:49.545 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:49.546 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-09-06T11:22:49.569 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:49.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-09-06T11:22:49.631 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:22:49.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-09-06T11:22:49.695 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd new d38834d4-e4b5-4fdc-acc4-9bf58e0737bc 0 2021-09-06T11:22:49.936 INFO:teuthology.orchestra.run.smithi094.stdout:0 2021-09-06T11:22:49.948 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd new 3924e060-8e16-43dc-adf8-7a0abe21fb98 1 2021-09-06T11:22:50.149 INFO:teuthology.orchestra.run.smithi094.stdout:1 2021-09-06T11:22:50.162 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd new 4752d8b8-eebb-4110-84a7-805bfe446938 2 2021-09-06T11:22:50.362 INFO:teuthology.orchestra.run.smithi094.stdout:2 2021-09-06T11:22:50.375 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-09-06T11:22:50.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-09-06T11:22:50.378 INFO:tasks.ceph.c1.osd.0:Started 2021-09-06T11:22:50.378 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-09-06T11:22:50.379 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-09-06T11:22:50.381 INFO:tasks.ceph.c1.osd.1:Started 2021-09-06T11:22:50.381 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-09-06T11:22:50.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-09-06T11:22:50.384 INFO:tasks.ceph.c1.osd.2:Started 2021-09-06T11:22:50.384 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-06T11:22:50.687 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-06T11:22:50.867 INFO:teuthology.misc.health.smithi094.stdout: 2021-09-06T11:22:50.868 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":5,"fsid":"dd8e0d3d-b2da-445f-9f1a-101a0b6989fd","created":"2021-09-06T11:22:49.429867+0000","modified":"2021-09-06T11:22:50.362488+0000","last_up_change":"0.000000","last_in_change":"2021-09-06T11:22:50.362488+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d38834d4-e4b5-4fdc-acc4-9bf58e0737bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3924e060-8e16-43dc-adf8-7a0abe21fb98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4752d8b8-eebb-4110-84a7-805bfe446938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T11:22:50.878 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-06T11:22:50.975 INFO:tasks.ceph.c1.osd.0.smithi094.stderr:2021-09-06T11:22:50.975+0000 7f9425bf20c0 -1 Falling back to public interface 2021-09-06T11:22:50.978 INFO:tasks.ceph.c1.osd.2.smithi094.stderr:2021-09-06T11:22:50.979+0000 7f783e1550c0 -1 Falling back to public interface 2021-09-06T11:22:50.979 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-09-06T11:22:50.979+0000 7fa03412b0c0 -1 Falling back to public interface 2021-09-06T11:22:50.981 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-09-06T11:22:50.981+0000 7fa02e2f0700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6818/0: (98) Address already in use 2021-09-06T11:22:53.723 INFO:tasks.ceph.c1.osd.0.smithi094.stderr:2021-09-06T11:22:53.722+0000 7f9425bf20c0 -1 osd.0 0 log_to_monitors {default=true} 2021-09-06T11:22:53.723 INFO:tasks.ceph.c1.osd.2.smithi094.stderr:2021-09-06T11:22:53.722+0000 7f783e1550c0 -1 osd.2 0 log_to_monitors {default=true} 2021-09-06T11:22:53.725 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-09-06T11:22:53.725+0000 7fa03412b0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-09-06T11:22:55.465 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-09-06T11:22:55.465+0000 7fa02aae9700 -1 osd.1 0 waiting for initial osdmap 2021-09-06T11:22:55.467 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-09-06T11:22:55.467+0000 7fa02347c700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T11:22:55.485 INFO:tasks.ceph.c1.osd.2.smithi094.stderr:2021-09-06T11:22:55.485+0000 7f7834b13700 -1 osd.2 0 waiting for initial osdmap 2021-09-06T11:22:55.487 INFO:tasks.ceph.c1.osd.2.smithi094.stderr:2021-09-06T11:22:55.487+0000 7f782d4a6700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T11:22:55.489 INFO:tasks.ceph.c1.osd.0.smithi094.stderr:2021-09-06T11:22:55.489+0000 7f941c5b0700 -1 osd.0 0 waiting for initial osdmap 2021-09-06T11:22:55.491 INFO:tasks.ceph.c1.osd.0.smithi094.stderr:2021-09-06T11:22:55.491+0000 7f9414f43700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T11:22:57.182 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-06T11:22:57.469 INFO:teuthology.misc.health.smithi094.stdout: 2021-09-06T11:22:57.470 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":9,"fsid":"dd8e0d3d-b2da-445f-9f1a-101a0b6989fd","created":"2021-09-06T11:22:49.429867+0000","modified":"2021-09-06T11:22:57.464673+0000","last_up_change":"2021-09-06T11:22:56.463951+0000","last_in_change":"2021-09-06T11:22:50.362488+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d38834d4-e4b5-4fdc-acc4-9bf58e0737bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":59460},{"type":"v1","addr":"172.21.15.94:6801","nonce":59460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":59460},{"type":"v1","addr":"172.21.15.94:6803","nonce":59460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":59460},{"type":"v1","addr":"172.21.15.94:6807","nonce":59460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":59460},{"type":"v1","addr":"172.21.15.94:6805","nonce":59460}]},"public_addr":"172.21.15.94:6801/59460","cluster_addr":"172.21.15.94:6803/59460","heartbeat_back_addr":"172.21.15.94:6807/59460","heartbeat_front_addr":"172.21.15.94:6805/59460","state":["exists","up"]},{"osd":1,"uuid":"3924e060-8e16-43dc-adf8-7a0abe21fb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6811","nonce":59461},{"type":"v1","addr":"172.21.15.94:6813","nonce":59461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":59461},{"type":"v1","addr":"172.21.15.94:6819","nonce":59461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":59461},{"type":"v1","addr":"172.21.15.94:6823","nonce":59461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":59461},{"type":"v1","addr":"172.21.15.94:6821","nonce":59461}]},"public_addr":"172.21.15.94:6813/59461","cluster_addr":"172.21.15.94:6819/59461","heartbeat_back_addr":"172.21.15.94:6823/59461","heartbeat_front_addr":"172.21.15.94:6821/59461","state":["exists","up"]},{"osd":2,"uuid":"4752d8b8-eebb-4110-84a7-805bfe446938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":59462},{"type":"v1","addr":"172.21.15.94:6809","nonce":59462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":59462},{"type":"v1","addr":"172.21.15.94:6812","nonce":59462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":59462},{"type":"v1","addr":"172.21.15.94:6818","nonce":59462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":59462},{"type":"v1","addr":"172.21.15.94:6815","nonce":59462}]},"public_addr":"172.21.15.94:6809/59462","cluster_addr":"172.21.15.94:6812/59462","heartbeat_back_addr":"172.21.15.94:6818/59462","heartbeat_front_addr":"172.21.15.94:6815/59462","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.769679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.766067+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.740581+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T11:22:57.481 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-06T11:22:57.481 INFO:tasks.ceph:Creating RBD pool 2021-09-06T11:22:57.482 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd pool create rbd 8 2021-09-06T11:22:58.473 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'rbd' created 2021-09-06T11:22:58.486 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-06T11:22:59.477 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-06T11:22:59.490 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-09-06T11:22:59.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-06T11:22:59.491 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-06T11:22:59.807 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:22:59.807 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":11,"fsid":"dd8e0d3d-b2da-445f-9f1a-101a0b6989fd","created":"2021-09-06T11:22:49.429867+0000","modified":"2021-09-06T11:22:59.466072+0000","last_up_change":"2021-09-06T11:22:56.463951+0000","last_in_change":"2021-09-06T11:22:50.362488+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-06T11:22:57.777926+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-06T11:22:58.321404+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d38834d4-e4b5-4fdc-acc4-9bf58e0737bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":59460},{"type":"v1","addr":"172.21.15.94:6801","nonce":59460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":59460},{"type":"v1","addr":"172.21.15.94:6803","nonce":59460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":59460},{"type":"v1","addr":"172.21.15.94:6807","nonce":59460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":59460},{"type":"v1","addr":"172.21.15.94:6805","nonce":59460}]},"public_addr":"172.21.15.94:6801/59460","cluster_addr":"172.21.15.94:6803/59460","heartbeat_back_addr":"172.21.15.94:6807/59460","heartbeat_front_addr":"172.21.15.94:6805/59460","state":["exists","up"]},{"osd":1,"uuid":"3924e060-8e16-43dc-adf8-7a0abe21fb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6811","nonce":59461},{"type":"v1","addr":"172.21.15.94:6813","nonce":59461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":59461},{"type":"v1","addr":"172.21.15.94:6819","nonce":59461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":59461},{"type":"v1","addr":"172.21.15.94:6823","nonce":59461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":59461},{"type":"v1","addr":"172.21.15.94:6821","nonce":59461}]},"public_addr":"172.21.15.94:6813/59461","cluster_addr":"172.21.15.94:6819/59461","heartbeat_back_addr":"172.21.15.94:6823/59461","heartbeat_front_addr":"172.21.15.94:6821/59461","state":["exists","up"]},{"osd":2,"uuid":"4752d8b8-eebb-4110-84a7-805bfe446938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":59462},{"type":"v1","addr":"172.21.15.94:6809","nonce":59462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":59462},{"type":"v1","addr":"172.21.15.94:6812","nonce":59462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":59462},{"type":"v1","addr":"172.21.15.94:6818","nonce":59462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":59462},{"type":"v1","addr":"172.21.15.94:6815","nonce":59462}]},"public_addr":"172.21.15.94:6809/59462","cluster_addr":"172.21.15.94:6812/59462","heartbeat_back_addr":"172.21.15.94:6818/59462","heartbeat_front_addr":"172.21.15.94:6815/59462","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.769679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.766067+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.740581+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T11:22:59.820 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-09-06T11:22:57.777926+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2021-09-06T11:22:58.321404+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2021-09-06T11:22:59.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-09-06T11:23:00.110 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 8 2021-09-06T11:23:00.123 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get .mgr pg_num 2021-09-06T11:23:00.407 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2021-09-06T11:23:00.420 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-09-06T11:23:00.420 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-09-06T11:23:00.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-09-06T11:23:00.746 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:00.758 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":59143},{"type":"v1","addr":"172.21.15.94:6825","nonce":59143}]},"active_addr":"172.21.15.94:6825/59143","active_change":"2021-09-06T11:22:54.289963+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2235930880}]},{"addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":1039618658}]},{"addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2434014063}]},{"addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":3341521875}]}]}} 2021-09-06T11:23:00.761 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-09-06T11:23:00.761 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-09-06T11:23:00.761 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-06T11:23:01.041 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:01.041 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":12,"fsid":"dd8e0d3d-b2da-445f-9f1a-101a0b6989fd","created":"2021-09-06T11:22:49.429867+0000","modified":"2021-09-06T11:23:00.466850+0000","last_up_change":"2021-09-06T11:22:56.463951+0000","last_in_change":"2021-09-06T11:22:50.362488+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-06T11:22:57.777926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-06T11:22:58.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d38834d4-e4b5-4fdc-acc4-9bf58e0737bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":59460},{"type":"v1","addr":"172.21.15.94:6801","nonce":59460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":59460},{"type":"v1","addr":"172.21.15.94:6803","nonce":59460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":59460},{"type":"v1","addr":"172.21.15.94:6807","nonce":59460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":59460},{"type":"v1","addr":"172.21.15.94:6805","nonce":59460}]},"public_addr":"172.21.15.94:6801/59460","cluster_addr":"172.21.15.94:6803/59460","heartbeat_back_addr":"172.21.15.94:6807/59460","heartbeat_front_addr":"172.21.15.94:6805/59460","state":["exists","up"]},{"osd":1,"uuid":"3924e060-8e16-43dc-adf8-7a0abe21fb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6811","nonce":59461},{"type":"v1","addr":"172.21.15.94:6813","nonce":59461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":59461},{"type":"v1","addr":"172.21.15.94:6819","nonce":59461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":59461},{"type":"v1","addr":"172.21.15.94:6823","nonce":59461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":59461},{"type":"v1","addr":"172.21.15.94:6821","nonce":59461}]},"public_addr":"172.21.15.94:6813/59461","cluster_addr":"172.21.15.94:6819/59461","heartbeat_back_addr":"172.21.15.94:6823/59461","heartbeat_front_addr":"172.21.15.94:6821/59461","state":["exists","up"]},{"osd":2,"uuid":"4752d8b8-eebb-4110-84a7-805bfe446938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":59462},{"type":"v1","addr":"172.21.15.94:6809","nonce":59462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":59462},{"type":"v1","addr":"172.21.15.94:6812","nonce":59462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":59462},{"type":"v1","addr":"172.21.15.94:6818","nonce":59462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":59462},{"type":"v1","addr":"172.21.15.94:6815","nonce":59462}]},"public_addr":"172.21.15.94:6809/59462","cluster_addr":"172.21.15.94:6812/59462","heartbeat_back_addr":"172.21.15.94:6818/59462","heartbeat_front_addr":"172.21.15.94:6815/59462","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.769679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.766067+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.740581+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T11:23:01.054 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-09-06T11:23:01.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-06T11:23:01.336 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:01.337 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":12,"fsid":"dd8e0d3d-b2da-445f-9f1a-101a0b6989fd","created":"2021-09-06T11:22:49.429867+0000","modified":"2021-09-06T11:23:00.466850+0000","last_up_change":"2021-09-06T11:22:56.463951+0000","last_in_change":"2021-09-06T11:22:50.362488+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-06T11:22:57.777926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-06T11:22:58.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d38834d4-e4b5-4fdc-acc4-9bf58e0737bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":59460},{"type":"v1","addr":"172.21.15.94:6801","nonce":59460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":59460},{"type":"v1","addr":"172.21.15.94:6803","nonce":59460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":59460},{"type":"v1","addr":"172.21.15.94:6807","nonce":59460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":59460},{"type":"v1","addr":"172.21.15.94:6805","nonce":59460}]},"public_addr":"172.21.15.94:6801/59460","cluster_addr":"172.21.15.94:6803/59460","heartbeat_back_addr":"172.21.15.94:6807/59460","heartbeat_front_addr":"172.21.15.94:6805/59460","state":["exists","up"]},{"osd":1,"uuid":"3924e060-8e16-43dc-adf8-7a0abe21fb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6811","nonce":59461},{"type":"v1","addr":"172.21.15.94:6813","nonce":59461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":59461},{"type":"v1","addr":"172.21.15.94:6819","nonce":59461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":59461},{"type":"v1","addr":"172.21.15.94:6823","nonce":59461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":59461},{"type":"v1","addr":"172.21.15.94:6821","nonce":59461}]},"public_addr":"172.21.15.94:6813/59461","cluster_addr":"172.21.15.94:6819/59461","heartbeat_back_addr":"172.21.15.94:6823/59461","heartbeat_front_addr":"172.21.15.94:6821/59461","state":["exists","up"]},{"osd":2,"uuid":"4752d8b8-eebb-4110-84a7-805bfe446938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":59462},{"type":"v1","addr":"172.21.15.94:6809","nonce":59462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":59462},{"type":"v1","addr":"172.21.15.94:6812","nonce":59462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":59462},{"type":"v1","addr":"172.21.15.94:6818","nonce":59462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":59462},{"type":"v1","addr":"172.21.15.94:6815","nonce":59462}]},"public_addr":"172.21.15.94:6809/59462","cluster_addr":"172.21.15.94:6812/59462","heartbeat_back_addr":"172.21.15.94:6818/59462","heartbeat_front_addr":"172.21.15.94:6815/59462","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.769679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.766067+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:22:54.740581+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T11:23:01.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-09-06T11:23:01.492 INFO:teuthology.orchestra.run.smithi094.stdout:34359738371 2021-09-06T11:23:01.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-09-06T11:23:01.637 INFO:teuthology.orchestra.run.smithi094.stdout:34359738371 2021-09-06T11:23:01.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-09-06T11:23:01.780 INFO:teuthology.orchestra.run.smithi094.stdout:34359738372 2021-09-06T11:23:01.781 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-06T11:23:02.060 INFO:teuthology.orchestra.run.smithi094.stdout:34359738370 2021-09-06T11:23:02.073 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2021-09-06T11:23:03.074 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-06T11:23:03.371 INFO:teuthology.orchestra.run.smithi094.stdout:34359738371 2021-09-06T11:23:03.384 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2021-09-06T11:23:03.385 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-09-06T11:23:03.665 INFO:teuthology.orchestra.run.smithi094.stdout:34359738371 2021-09-06T11:23:03.678 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2021-09-06T11:23:03.679 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-09-06T11:23:03.961 INFO:teuthology.orchestra.run.smithi094.stdout:34359738372 2021-09-06T11:23:03.974 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2021-09-06T11:23:03.975 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-09-06T11:23:03.975 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-06T11:23:04.250 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2021-09-06T11:23:04.262 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-09-06T11:23:02.294158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24832,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23104,"kb_avail":314547968,"statfs":{"total":322122547200,"available":322097119232,"internally_reserved":0,"allocated":1482752,"data_stored":1256997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23658496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.473816+0000","last_change":"2021-09-06T11:22:59.473816+0000","last_active":"2021-09-06T11:22:59.473816+0000","last_peered":"2021-09-06T11:22:59.473816+0000","last_clean":"2021-09-06T11:22:59.473816+0000","last_became_active":"2021-09-06T11:22:59.473528+0000","last_became_peered":"2021-09-06T11:22:59.473528+0000","last_unstale":"2021-09-06T11:22:59.473816+0000","last_undegraded":"2021-09-06T11:22:59.473816+0000","last_fullsized":"2021-09-06T11:22:59.473816+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.472453+0000","last_change":"2021-09-06T11:22:59.472453+0000","last_active":"2021-09-06T11:22:59.472453+0000","last_peered":"2021-09-06T11:22:59.472453+0000","last_clean":"2021-09-06T11:22:59.472453+0000","last_became_active":"2021-09-06T11:22:59.472039+0000","last_became_peered":"2021-09-06T11:22:59.472039+0000","last_unstale":"2021-09-06T11:22:59.472453+0000","last_undegraded":"2021-09-06T11:22:59.472453+0000","last_fullsized":"2021-09-06T11:22:59.472453+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.476120+0000","last_change":"2021-09-06T11:22:59.475192+0000","last_active":"2021-09-06T11:22:59.476120+0000","last_peered":"2021-09-06T11:22:59.476120+0000","last_clean":"2021-09-06T11:22:59.476120+0000","last_became_active":"2021-09-06T11:22:59.474141+0000","last_became_peered":"2021-09-06T11:22:59.474141+0000","last_unstale":"2021-09-06T11:22:59.476120+0000","last_undegraded":"2021-09-06T11:22:59.476120+0000","last_fullsized":"2021-09-06T11:22:59.476120+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.472330+0000","last_change":"2021-09-06T11:22:59.472330+0000","last_active":"2021-09-06T11:22:59.472330+0000","last_peered":"2021-09-06T11:22:59.472330+0000","last_clean":"2021-09-06T11:22:59.472330+0000","last_became_active":"2021-09-06T11:22:59.471995+0000","last_became_peered":"2021-09-06T11:22:59.471995+0000","last_unstale":"2021-09-06T11:22:59.472330+0000","last_undegraded":"2021-09-06T11:22:59.472330+0000","last_fullsized":"2021-09-06T11:22:59.472330+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.473063+0000","last_change":"2021-09-06T11:22:59.473063+0000","last_active":"2021-09-06T11:22:59.473063+0000","last_peered":"2021-09-06T11:22:59.473063+0000","last_clean":"2021-09-06T11:22:59.473063+0000","last_became_active":"2021-09-06T11:22:59.472764+0000","last_became_peered":"2021-09-06T11:22:59.472764+0000","last_unstale":"2021-09-06T11:22:59.473063+0000","last_undegraded":"2021-09-06T11:22:59.473063+0000","last_fullsized":"2021-09-06T11:22:59.473063+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.922921+0000","last_change":"2021-09-06T11:22:59.475039+0000","last_active":"2021-09-06T11:22:59.922921+0000","last_peered":"2021-09-06T11:22:59.922921+0000","last_clean":"2021-09-06T11:22:59.922921+0000","last_became_active":"2021-09-06T11:22:59.473134+0000","last_became_peered":"2021-09-06T11:22:59.473134+0000","last_unstale":"2021-09-06T11:22:59.922921+0000","last_undegraded":"2021-09-06T11:22:59.922921+0000","last_fullsized":"2021-09-06T11:22:59.922921+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.475158+0000","last_change":"2021-09-06T11:22:59.475158+0000","last_active":"2021-09-06T11:22:59.475158+0000","last_peered":"2021-09-06T11:22:59.475158+0000","last_clean":"2021-09-06T11:22:59.475158+0000","last_became_active":"2021-09-06T11:22:59.474778+0000","last_became_peered":"2021-09-06T11:22:59.474778+0000","last_unstale":"2021-09-06T11:22:59.475158+0000","last_undegraded":"2021-09-06T11:22:59.475158+0000","last_fullsized":"2021-09-06T11:22:59.475158+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.471733+0000","last_change":"2021-09-06T11:22:59.471733+0000","last_active":"2021-09-06T11:22:59.471733+0000","last_peered":"2021-09-06T11:22:59.471733+0000","last_clean":"2021-09-06T11:22:59.471733+0000","last_became_active":"2021-09-06T11:22:59.471433+0000","last_became_peered":"2021-09-06T11:22:59.471433+0000","last_unstale":"2021-09-06T11:22:59.471733+0000","last_undegraded":"2021-09-06T11:22:59.471733+0000","last_fullsized":"2021-09-06T11:22:59.471733+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.476736+0000","last_change":"2021-09-06T11:22:59.474895+0000","last_active":"2021-09-06T11:22:59.476736+0000","last_peered":"2021-09-06T11:22:59.476736+0000","last_clean":"2021-09-06T11:22:59.476736+0000","last_became_active":"2021-09-06T11:22:59.474397+0000","last_became_peered":"2021-09-06T11:22:59.474397+0000","last_unstale":"2021-09-06T11:22:59.476736+0000","last_undegraded":"2021-09-06T11:22:59.476736+0000","last_fullsized":"2021-09-06T11:22:59.476736+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8624,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104848976,"statfs":{"total":107374182400,"available":107365351424,"internally_reserved":0,"allocated":696320,"data_stored":618012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7528,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7424,"kb_avail":104850072,"statfs":{"total":107374182400,"available":107366473728,"internally_reserved":0,"allocated":98304,"data_stored":24777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7602176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8680,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848920,"statfs":{"total":107374182400,"available":107365294080,"internally_reserved":0,"allocated":688128,"data_stored":614208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-06T11:23:04.264 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-06T11:23:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2021-09-06T11:23:04.554 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2021-09-06T11:23:04.294643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24832,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23104,"kb_avail":314547968,"statfs":{"total":322122547200,"available":322097119232,"internally_reserved":0,"allocated":1482752,"data_stored":1256997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23658496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000484"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.473816+0000","last_change":"2021-09-06T11:22:59.473816+0000","last_active":"2021-09-06T11:22:59.473816+0000","last_peered":"2021-09-06T11:22:59.473816+0000","last_clean":"2021-09-06T11:22:59.473816+0000","last_became_active":"2021-09-06T11:22:59.473528+0000","last_became_peered":"2021-09-06T11:22:59.473528+0000","last_unstale":"2021-09-06T11:22:59.473816+0000","last_undegraded":"2021-09-06T11:22:59.473816+0000","last_fullsized":"2021-09-06T11:22:59.473816+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.472453+0000","last_change":"2021-09-06T11:22:59.472453+0000","last_active":"2021-09-06T11:22:59.472453+0000","last_peered":"2021-09-06T11:22:59.472453+0000","last_clean":"2021-09-06T11:22:59.472453+0000","last_became_active":"2021-09-06T11:22:59.472039+0000","last_became_peered":"2021-09-06T11:22:59.472039+0000","last_unstale":"2021-09-06T11:22:59.472453+0000","last_undegraded":"2021-09-06T11:22:59.472453+0000","last_fullsized":"2021-09-06T11:22:59.472453+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.476120+0000","last_change":"2021-09-06T11:22:59.475192+0000","last_active":"2021-09-06T11:22:59.476120+0000","last_peered":"2021-09-06T11:22:59.476120+0000","last_clean":"2021-09-06T11:22:59.476120+0000","last_became_active":"2021-09-06T11:22:59.474141+0000","last_became_peered":"2021-09-06T11:22:59.474141+0000","last_unstale":"2021-09-06T11:22:59.476120+0000","last_undegraded":"2021-09-06T11:22:59.476120+0000","last_fullsized":"2021-09-06T11:22:59.476120+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.472330+0000","last_change":"2021-09-06T11:22:59.472330+0000","last_active":"2021-09-06T11:22:59.472330+0000","last_peered":"2021-09-06T11:22:59.472330+0000","last_clean":"2021-09-06T11:22:59.472330+0000","last_became_active":"2021-09-06T11:22:59.471995+0000","last_became_peered":"2021-09-06T11:22:59.471995+0000","last_unstale":"2021-09-06T11:22:59.472330+0000","last_undegraded":"2021-09-06T11:22:59.472330+0000","last_fullsized":"2021-09-06T11:22:59.472330+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.473063+0000","last_change":"2021-09-06T11:22:59.473063+0000","last_active":"2021-09-06T11:22:59.473063+0000","last_peered":"2021-09-06T11:22:59.473063+0000","last_clean":"2021-09-06T11:22:59.473063+0000","last_became_active":"2021-09-06T11:22:59.472764+0000","last_became_peered":"2021-09-06T11:22:59.472764+0000","last_unstale":"2021-09-06T11:22:59.473063+0000","last_undegraded":"2021-09-06T11:22:59.473063+0000","last_fullsized":"2021-09-06T11:22:59.473063+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.922921+0000","last_change":"2021-09-06T11:22:59.475039+0000","last_active":"2021-09-06T11:22:59.922921+0000","last_peered":"2021-09-06T11:22:59.922921+0000","last_clean":"2021-09-06T11:22:59.922921+0000","last_became_active":"2021-09-06T11:22:59.473134+0000","last_became_peered":"2021-09-06T11:22:59.473134+0000","last_unstale":"2021-09-06T11:22:59.922921+0000","last_undegraded":"2021-09-06T11:22:59.922921+0000","last_fullsized":"2021-09-06T11:22:59.922921+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.475158+0000","last_change":"2021-09-06T11:22:59.475158+0000","last_active":"2021-09-06T11:22:59.475158+0000","last_peered":"2021-09-06T11:22:59.475158+0000","last_clean":"2021-09-06T11:22:59.475158+0000","last_became_active":"2021-09-06T11:22:59.474778+0000","last_became_peered":"2021-09-06T11:22:59.474778+0000","last_unstale":"2021-09-06T11:22:59.475158+0000","last_undegraded":"2021-09-06T11:22:59.475158+0000","last_fullsized":"2021-09-06T11:22:59.475158+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.471733+0000","last_change":"2021-09-06T11:22:59.471733+0000","last_active":"2021-09-06T11:22:59.471733+0000","last_peered":"2021-09-06T11:22:59.471733+0000","last_clean":"2021-09-06T11:22:59.471733+0000","last_became_active":"2021-09-06T11:22:59.471433+0000","last_became_peered":"2021-09-06T11:22:59.471433+0000","last_unstale":"2021-09-06T11:22:59.471733+0000","last_undegraded":"2021-09-06T11:22:59.471733+0000","last_fullsized":"2021-09-06T11:22:59.471733+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T11:22:59.476736+0000","last_change":"2021-09-06T11:22:59.474895+0000","last_active":"2021-09-06T11:22:59.476736+0000","last_peered":"2021-09-06T11:22:59.476736+0000","last_clean":"2021-09-06T11:22:59.476736+0000","last_became_active":"2021-09-06T11:22:59.474397+0000","last_became_peered":"2021-09-06T11:22:59.474397+0000","last_unstale":"2021-09-06T11:22:59.476736+0000","last_undegraded":"2021-09-06T11:22:59.476736+0000","last_fullsized":"2021-09-06T11:22:59.476736+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:22:58.465223+0000","last_clean_scrub_stamp":"2021-09-06T11:22:58.465223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8624,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104848976,"statfs":{"total":107374182400,"available":107365351424,"internally_reserved":0,"allocated":696320,"data_stored":618012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7528,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7424,"kb_avail":104850072,"statfs":{"total":107374182400,"available":107366473728,"internally_reserved":0,"allocated":98304,"data_stored":24777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7602176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8680,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848920,"statfs":{"total":107374182400,"available":107365294080,"internally_reserved":0,"allocated":688128,"data_stored":614208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-06T11:23:04.555 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-09-06T11:23:04.555 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-09-06T11:23:04.556 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-09-06T11:23:04.556 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-09-06T11:23:04.896 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:04.897 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-06T11:23:04.909 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-09-06T11:23:04.910 INFO:teuthology.run_tasks:Running task ceph... 2021-09-06T11:23:04.920 INFO:tasks.ceph:Creating ceph cluster c2... 2021-09-06T11:23:04.921 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-06T11:23:04.921 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi/6376787', 'branch': 'wip-skoduri-dbstore-vstart', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6376787', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'soumyakoduri-2021-09-06_10:28:39-rgw-wip-skoduri-dbstore-vstart-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.3', 'overrides': {'admin_socket': {'branch': 'wip-skoduri-dbstore-vstart'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-skoduri-dbstore-vstart', 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e'}}, 'owner': 'scheduled_soumyakoduri@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-skoduri-dbstore-vstart', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_2227f2382f1df5117a62687acd2d99ea5eeb871e/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'targets': {'smithi094.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUTN53Z3n5QlqTDgk0NpEVg01dFbGUi6kCFNTkBAjTvi+FP0HPPz4m/nWJJWMgn9J9x87EXF9d5j4yhDZZUbFiKnCtE0dFqp7yDt+646ytY2pcOWqdnv3fMuK2YhdkRApWI6PATZkxRTQwnfDUxTWfHFjtREnNxDmZfc2W0YrsU3gCqByrMTHrycORc7y3OUyenla28nMmmYOIRAOJmwJi5dZBtxSK85jeYMra3DXk/yW3puzjQiyWMtUhdo4xEE/dnf9rAENBmVX7dCyo1Kei73l5gzG0b5dlipQcx3coHbU+P6YpHmRbxYRNFGX2LKiW2Qe9N0tS4EYmfK2kRsvcurg9UYzOof3BHPHYDAFygcudl9WTELEmtRTOYXltluuzCFxpnO6S8z+pfnExOIx9aPCm0dvzJlQyfOTFs+OggDmYQOPlQnoiZLEbfukHyPaig4GpWwscWbmWsOPTqwF2PPyOoATGx5+xfcHkmYmM26ahbjySb/iMipmGwvdWwZU=', 'smithi139.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxOSAXu0ahF2fafU0XYIhsUwAZYZNMc0rU5nwzbFpNLJIohFw7h/mcnGZ+vhFkODxC9wceI+y+q0Ku77hekOeNq2jhxMbe9ursZc16VNyEGSg8FYaJQ/pxeHtRGF7kpKP6Tjq1qTHnRIUu+8TuWjnw7F85vyH64WvQORUNGmeH3OEz3lv+ZQGFLO5v2w1Vux/q42+sYA6KGNSFqXGfYDha3W3HUmS+d+K/z6ZIz1ZUGN4eCikpM2wL9gcirFUOKnCLxjgrDI9cTZY01z21PQG4HeYoaH7ZTFO2rWVkzAZLPhhHVRNzgLJe1fIuAcFG3GOplFJI4GEWg+VOQNsh4+SG1Mm1rrWZSYrwQ74bNDrFD8kCnLfNcKqpo0t2Qh1IqbPOAcrWoAqnoMFzOlh1kkc0WsF30uEhQVnHkIlF0yZfWhKuHx0gwG9+jotBkxUImCL1Pj1hzupFeFIMdhVY7PLurYFJtwqvKFXl7WeWoub2qHGq7S6tbv6oqxQlksp2M/E='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2227f2382f1df5117a62687acd2d99ea5eeb871e', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-09-06_10:28:39', 'tube': 'smithi', 'user': 'soumyakoduri', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-06T11:23:04.922 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-06T11:23:04.925 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-06T11:23:04.942 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2021-09-06T11:23:04.979 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/run/ceph 2021-09-06T11:23:05.011 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:05.011 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2021-09-06T11:23:05.068 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-06T11:23:05.068 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-06T11:23:05.124 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-06T11:23:05.124 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T11:23:05.125 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 138512 Links: 1 2021-09-06T11:23:05.125 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T11:23:05.125 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T11:23:05.125 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2021-09-06 11:22:34.875985963 +0000 2021-09-06T11:23:05.126 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2021-09-06 11:03:53.709040802 +0000 2021-09-06T11:23:05.126 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2021-09-06 11:03:53.709040802 +0000 2021-09-06T11:23:05.126 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2021-09-06T11:23:05.135 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2021-09-06T11:23:05.135 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2021-09-06T11:23:05.136 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000319938 s, 1.6 MB/s 2021-09-06T11:23:05.139 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-06T11:23:05.195 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-06T11:23:05.195 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T11:23:05.195 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 135012 Links: 1 2021-09-06T11:23:05.195 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T11:23:05.196 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T11:23:05.196 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2021-09-06 11:22:34.875985963 +0000 2021-09-06T11:23:05.196 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2021-09-06 11:03:54.052033505 +0000 2021-09-06T11:23:05.196 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2021-09-06 11:03:54.052033505 +0000 2021-09-06T11:23:05.197 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2021-09-06T11:23:05.205 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2021-09-06T11:23:05.206 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2021-09-06T11:23:05.206 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.0001955 s, 2.6 MB/s 2021-09-06T11:23:05.208 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-06T11:23:05.264 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-06T11:23:05.264 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T11:23:05.265 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 133975 Links: 1 2021-09-06T11:23:05.265 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T11:23:05.265 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T11:23:05.265 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2021-09-06 11:22:34.875985963 +0000 2021-09-06T11:23:05.266 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2021-09-06 11:03:54.390026313 +0000 2021-09-06T11:23:05.266 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2021-09-06 11:03:54.390026313 +0000 2021-09-06T11:23:05.266 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2021-09-06T11:23:05.274 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2021-09-06T11:23:05.274 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2021-09-06T11:23:05.275 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000240758 s, 2.1 MB/s 2021-09-06T11:23:05.277 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-06T11:23:05.333 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-06T11:23:05.333 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T11:23:05.333 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 137750 Links: 1 2021-09-06T11:23:05.334 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T11:23:05.334 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T11:23:05.334 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2021-09-06 11:22:34.875985963 +0000 2021-09-06T11:23:05.334 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2021-09-06 11:03:54.729019101 +0000 2021-09-06T11:23:05.335 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2021-09-06 11:03:54.729019101 +0000 2021-09-06T11:23:05.335 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2021-09-06T11:23:05.342 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2021-09-06T11:23:05.343 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2021-09-06T11:23:05.343 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000206224 s, 2.5 MB/s 2021-09-06T11:23:05.345 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T11:23:05.346 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi139.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-06T11:23:05.346 INFO:tasks.ceph:Generating config... 2021-09-06T11:23:05.347 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-06T11:23:05.348 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-06T11:23:05.348 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-06T11:23:05.348 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T11:23:05.349 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-06T11:23:05.349 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-06T11:23:05.349 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-06T11:23:05.350 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-06T11:23:05.350 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-06T11:23:05.350 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-06T11:23:05.351 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-06T11:23:05.351 INFO:tasks.ceph:[client] setuser = ceph 2021-09-06T11:23:05.351 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-06T11:23:05.352 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-06T11:23:05.352 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-06T11:23:05.352 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-06T11:23:05.352 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-06T11:23:05.353 INFO:tasks.ceph:[osd] debug filestore = 20 2021-09-06T11:23:05.353 INFO:tasks.ceph:[osd] debug journal = 20 2021-09-06T11:23:05.353 INFO:tasks.ceph:[osd] debug ms = 20 2021-09-06T11:23:05.354 INFO:tasks.ceph:[osd] debug osd = 25 2021-09-06T11:23:05.354 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-06T11:23:05.354 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-06T11:23:05.355 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-06T11:23:05.355 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-06T11:23:05.355 INFO:tasks.ceph:Setting up c2.mon.a... 2021-09-06T11:23:05.356 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-09-06T11:23:05.455 INFO:teuthology.orchestra.run.smithi139.stdout:creating /etc/ceph/c2.keyring 2021-09-06T11:23:05.457 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-09-06T11:23:05.496 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-06T11:23:05.561 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.139')] 2021-09-06T11:23:05.561 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.139'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-06T11:23:05.562 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:05.563 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-06T11:23:05.615 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.139 --print /home/ubuntu/cephtest/c2.monmap 2021-09-06T11:23:05.691 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:23:05.691 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:23:05.692 INFO:teuthology.orchestra.run.smithi139.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-09-06T11:23:05.693 INFO:teuthology.orchestra.run.smithi139.stdout:monmaptool: generated fsid 4178eb40-61c1-44a5-9987-53bfc91286c0 2021-09-06T11:23:05.693 INFO:teuthology.orchestra.run.smithi139.stdout:setting min_mon_release = octopus 2021-09-06T11:23:05.693 INFO:teuthology.orchestra.run.smithi139.stdout:epoch 0 2021-09-06T11:23:05.693 INFO:teuthology.orchestra.run.smithi139.stdout:fsid 4178eb40-61c1-44a5-9987-53bfc91286c0 2021-09-06T11:23:05.694 INFO:teuthology.orchestra.run.smithi139.stdout:last_changed 2021-09-06T11:23:05.692307+0000 2021-09-06T11:23:05.694 INFO:teuthology.orchestra.run.smithi139.stdout:created 2021-09-06T11:23:05.692307+0000 2021-09-06T11:23:05.694 INFO:teuthology.orchestra.run.smithi139.stdout:min_mon_release 15 (octopus) 2021-09-06T11:23:05.694 INFO:teuthology.orchestra.run.smithi139.stdout:election_strategy: 1 2021-09-06T11:23:05.695 INFO:teuthology.orchestra.run.smithi139.stdout:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2021-09-06T11:23:05.695 INFO:teuthology.orchestra.run.smithi139.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-09-06T11:23:05.695 DEBUG:teuthology.orchestra.run.smithi139:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-06T11:23:05.747 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 4178eb40-61c1-44a5-9987-53bfc91286c0... 2021-09-06T11:23:05.748 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-06T11:23:05.751 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-06T11:23:05.797 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2021-09-06T11:23:05.798 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2021-09-06T11:23:05.798 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-06T11:23:05.798 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2021-09-06T11:23:05.798 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.799 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2021-09-06T11:23:05.799 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.799 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2021-09-06T11:23:05.799 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.799 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2021-09-06T11:23:05.800 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2021-09-06T11:23:05.800 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.800 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2021-09-06T11:23:05.800 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2021-09-06T11:23:05.801 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.801 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-06T11:23:05.801 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2021-09-06T11:23:05.801 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.801 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2021-09-06T11:23:05.802 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-06T11:23:05.802 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.802 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2021-09-06T11:23:05.802 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.802 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2021-09-06T11:23:05.803 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2021-09-06T11:23:05.803 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2021-09-06T11:23:05.803 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2021-09-06T11:23:05.803 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2021-09-06T11:23:05.804 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2021-09-06T11:23:05.804 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2021-09-06T11:23:05.804 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-06T11:23:05.804 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-06T11:23:05.804 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2021-09-06T11:23:05.805 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.805 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-06T11:23:05.805 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.805 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2021-09-06T11:23:05.805 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.806 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2021-09-06T11:23:05.806 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.806 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2021-09-06T11:23:05.806 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2021-09-06T11:23:05.807 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2021-09-06T11:23:05.807 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = 172.21.15.139 2021-09-06T11:23:05.807 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 4178eb40-61c1-44a5-9987-53bfc91286c0 2021-09-06T11:23:05.807 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.807 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2021-09-06T11:23:05.808 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2021-09-06T11:23:05.808 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.808 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2021-09-06T11:23:05.808 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2021-09-06T11:23:05.808 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.809 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2021-09-06T11:23:05.809 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2021-09-06T11:23:05.809 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.809 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2021-09-06T11:23:05.810 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2021-09-06T11:23:05.810 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2021-09-06T11:23:05.810 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.810 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2021-09-06T11:23:05.810 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2021-09-06T11:23:05.811 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.811 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2021-09-06T11:23:05.811 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.811 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2021-09-06T11:23:05.811 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.812 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2021-09-06T11:23:05.812 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.812 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2021-09-06T11:23:05.812 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2021-09-06T11:23:05.812 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.813 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2021-09-06T11:23:05.813 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2021-09-06T11:23:05.813 INFO:teuthology.orchestra.run.smithi094.stdout: debug filestore = 20 2021-09-06T11:23:05.813 INFO:teuthology.orchestra.run.smithi094.stdout: debug journal = 20 2021-09-06T11:23:05.814 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 20 2021-09-06T11:23:05.814 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 25 2021-09-06T11:23:05.814 INFO:teuthology.orchestra.run.smithi094.stdout: rocksdb delete range threshold = 0 2021-09-06T11:23:05.814 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.814 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2021-09-06T11:23:05.815 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2021-09-06T11:23:05.815 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2021-09-06T11:23:05.815 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2021-09-06T11:23:05.815 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2021-09-06T11:23:05.815 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2021-09-06T11:23:05.816 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2021-09-06T11:23:05.816 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2021-09-06T11:23:05.816 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.816 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2021-09-06T11:23:05.817 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2021-09-06T11:23:05.817 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2021-09-06T11:23:05.817 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2021-09-06T11:23:05.817 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2021-09-06T11:23:05.817 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2021-09-06T11:23:05.818 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2021-09-06T11:23:05.818 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2021-09-06T11:23:05.818 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2021-09-06T11:23:05.818 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2021-09-06T11:23:05.819 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2021-09-06T11:23:05.819 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.819 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-06T11:23:05.819 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2021-09-06T11:23:05.819 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2021-09-06T11:23:05.820 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.820 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2021-09-06T11:23:05.820 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-06T11:23:05.820 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-06T11:23:05.820 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:23:05.821 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2021-09-06T11:23:05.821 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2021-09-06T11:23:05.821 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2021-09-06T11:23:05.821 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2021-09-06T11:23:05.822 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-06T11:23:05.822 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-06T11:23:05.822 INFO:teuthology.orchestra.run.smithi094.stdout: debug rgw = 20 2021-09-06T11:23:05.822 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt require ssl = False 2021-09-06T11:23:05.822 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt s3 kms backend = testing 2021-09-06T11:23:05.823 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T11:23:05.823 INFO:teuthology.orchestra.run.smithi094.stdout: rgw curl low speed time = 300 2021-09-06T11:23:05.823 INFO:teuthology.orchestra.run.smithi094.stdout: rgw data log num shards = 4 2021-09-06T11:23:05.823 INFO:teuthology.orchestra.run.smithi094.stdout: rgw md log max shards = 4 2021-09-06T11:23:05.823 INFO:teuthology.orchestra.run.smithi094.stdout: rgw sync log trim interval = 0 2021-09-06T11:23:05.824 INFO:teuthology.orchestra.run.smithi094.stdout: rgw sync meta inject err probability = 0.1 2021-09-06T11:23:05.824 INFO:teuthology.orchestra.run.smithi094.stdout: rgw sync obj etag verify = True 2021-09-06T11:23:05.824 INFO:teuthology.orchestra.run.smithi094.stdout: setgroup = ceph 2021-09-06T11:23:05.824 INFO:teuthology.orchestra.run.smithi094.stdout: setuser = ceph 2021-09-06T11:23:05.825 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2021-09-06T11:23:05.825 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2021-09-06T11:23:05.825 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 10 2021-09-06T11:23:05.825 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2021-09-06T11:23:05.825 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 1000 2021-09-06T11:23:05.826 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2021-09-06T11:23:05.826 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 10000 2021-09-06T11:23:05.827 INFO:teuthology.orchestra.run.smithi139.stdout:[global] 2021-09-06T11:23:05.827 INFO:teuthology.orchestra.run.smithi139.stdout: chdir = "" 2021-09-06T11:23:05.828 INFO:teuthology.orchestra.run.smithi139.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-06T11:23:05.828 INFO:teuthology.orchestra.run.smithi139.stdout: auth supported = cephx 2021-09-06T11:23:05.828 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.828 INFO:teuthology.orchestra.run.smithi139.stdout: filestore xattr use omap = true 2021-09-06T11:23:05.829 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.829 INFO:teuthology.orchestra.run.smithi139.stdout: mon clock drift allowed = 1.000 2021-09-06T11:23:05.829 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.829 INFO:teuthology.orchestra.run.smithi139.stdout: osd crush chooseleaf type = 0 2021-09-06T11:23:05.829 INFO:teuthology.orchestra.run.smithi139.stdout: auth debug = true 2021-09-06T11:23:05.830 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.830 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on old message = true 2021-09-06T11:23:05.830 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on bug = true 2021-09-06T11:23:05.830 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.831 INFO:teuthology.orchestra.run.smithi139.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-06T11:23:05.831 INFO:teuthology.orchestra.run.smithi139.stdout: mon pg warn max object skew = 0 2021-09-06T11:23:05.831 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.831 INFO:teuthology.orchestra.run.smithi139.stdout: # disable pg_autoscaler by default for new pools 2021-09-06T11:23:05.831 INFO:teuthology.orchestra.run.smithi139.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-06T11:23:05.832 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.832 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default size = 2 2021-09-06T11:23:05.832 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.832 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow primary affinity = true 2021-09-06T11:23:05.832 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow pg remap = true 2021-09-06T11:23:05.833 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on legacy crush tunables = false 2021-09-06T11:23:05.833 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on crush straw calc version zero = false 2021-09-06T11:23:05.833 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on no sortbitwise = false 2021-09-06T11:23:05.833 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on osd down out interval zero = false 2021-09-06T11:23:05.834 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on too few osds = false 2021-09-06T11:23:05.834 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-06T11:23:05.834 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-06T11:23:05.834 INFO:teuthology.orchestra.run.smithi139.stdout: mon_allow_pool_size_one = true 2021-09-06T11:23:05.834 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.835 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-06T11:23:05.835 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.835 INFO:teuthology.orchestra.run.smithi139.stdout: osd default data pool replay window = 5 2021-09-06T11:23:05.835 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.835 INFO:teuthology.orchestra.run.smithi139.stdout: mon allow pool delete = true 2021-09-06T11:23:05.836 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.836 INFO:teuthology.orchestra.run.smithi139.stdout: mon cluster log file level = debug 2021-09-06T11:23:05.836 INFO:teuthology.orchestra.run.smithi139.stdout: debug asserts on shutdown = true 2021-09-06T11:23:05.836 INFO:teuthology.orchestra.run.smithi139.stdout: mon health detail to clog = false 2021-09-06T11:23:05.837 INFO:teuthology.orchestra.run.smithi139.stdout: mon host = 172.21.15.139 2021-09-06T11:23:05.837 INFO:teuthology.orchestra.run.smithi139.stdout: fsid = 4178eb40-61c1-44a5-9987-53bfc91286c0 2021-09-06T11:23:05.837 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.837 INFO:teuthology.orchestra.run.smithi139.stdout:[osd] 2021-09-06T11:23:05.837 INFO:teuthology.orchestra.run.smithi139.stdout: osd journal size = 100 2021-09-06T11:23:05.838 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.838 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub load threshold = 5.0 2021-09-06T11:23:05.838 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub max interval = 600 2021-09-06T11:23:05.838 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.839 INFO:teuthology.orchestra.run.smithi139.stdout: osd recover clone overlap = true 2021-09-06T11:23:05.839 INFO:teuthology.orchestra.run.smithi139.stdout: osd recovery max chunk = 1048576 2021-09-06T11:23:05.839 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.839 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug shutdown = true 2021-09-06T11:23:05.839 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug op order = true 2021-09-06T11:23:05.840 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify stray on activate = true 2021-09-06T11:23:05.840 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.840 INFO:teuthology.orchestra.run.smithi139.stdout: osd open classes on start = true 2021-09-06T11:23:05.840 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug pg log writeout = true 2021-09-06T11:23:05.840 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.841 INFO:teuthology.orchestra.run.smithi139.stdout: osd deep scrub update digest min age = 30 2021-09-06T11:23:05.841 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.841 INFO:teuthology.orchestra.run.smithi139.stdout: osd map max advance = 10 2021-09-06T11:23:05.841 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.841 INFO:teuthology.orchestra.run.smithi139.stdout: journal zero on create = true 2021-09-06T11:23:05.842 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.842 INFO:teuthology.orchestra.run.smithi139.stdout: filestore ondisk finisher threads = 3 2021-09-06T11:23:05.842 INFO:teuthology.orchestra.run.smithi139.stdout: filestore apply finisher threads = 3 2021-09-06T11:23:05.842 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.843 INFO:teuthology.orchestra.run.smithi139.stdout: bdev debug aio = true 2021-09-06T11:23:05.843 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug misdirected ops = true 2021-09-06T11:23:05.843 INFO:teuthology.orchestra.run.smithi139.stdout: debug filestore = 20 2021-09-06T11:23:05.843 INFO:teuthology.orchestra.run.smithi139.stdout: debug journal = 20 2021-09-06T11:23:05.843 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 20 2021-09-06T11:23:05.844 INFO:teuthology.orchestra.run.smithi139.stdout: debug osd = 25 2021-09-06T11:23:05.844 INFO:teuthology.orchestra.run.smithi139.stdout: rocksdb delete range threshold = 0 2021-09-06T11:23:05.844 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.844 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr] 2021-09-06T11:23:05.844 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2021-09-06T11:23:05.845 INFO:teuthology.orchestra.run.smithi139.stdout: debug mgr = 20 2021-09-06T11:23:05.845 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2021-09-06T11:23:05.845 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2021-09-06T11:23:05.845 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2021-09-06T11:23:05.845 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2021-09-06T11:23:05.846 INFO:teuthology.orchestra.run.smithi139.stdout: mgr/telemetry/nag = false 2021-09-06T11:23:05.846 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.846 INFO:teuthology.orchestra.run.smithi139.stdout:[mon] 2021-09-06T11:23:05.846 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2021-09-06T11:23:05.847 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2021-09-06T11:23:05.847 INFO:teuthology.orchestra.run.smithi139.stdout: debug paxos = 20 2021-09-06T11:23:05.847 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2021-09-06T11:23:05.847 INFO:teuthology.orchestra.run.smithi139.stdout: mon data avail warn = 5 2021-09-06T11:23:05.847 INFO:teuthology.orchestra.run.smithi139.stdout: mon mgr mkfs grace = 240 2021-09-06T11:23:05.848 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2021-09-06T11:23:05.848 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd reporter subtree level = osd 2021-09-06T11:23:05.848 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd prime pg temp = true 2021-09-06T11:23:05.848 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2021-09-06T11:23:05.848 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.849 INFO:teuthology.orchestra.run.smithi139.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-06T11:23:05.849 INFO:teuthology.orchestra.run.smithi139.stdout: auth mon ticket ttl = 660 # 11m 2021-09-06T11:23:05.849 INFO:teuthology.orchestra.run.smithi139.stdout: auth service ticket ttl = 240 # 4m 2021-09-06T11:23:05.849 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.849 INFO:teuthology.orchestra.run.smithi139.stdout: # don't complain about insecure global_id in the test suite 2021-09-06T11:23:05.850 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-06T11:23:05.850 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-06T11:23:05.850 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:05.850 INFO:teuthology.orchestra.run.smithi139.stdout:[client] 2021-09-06T11:23:05.851 INFO:teuthology.orchestra.run.smithi139.stdout: rgw cache enabled = true 2021-09-06T11:23:05.851 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable ops log = true 2021-09-06T11:23:05.851 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable usage log = true 2021-09-06T11:23:05.851 INFO:teuthology.orchestra.run.smithi139.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-06T11:23:05.851 INFO:teuthology.orchestra.run.smithi139.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-06T11:23:05.852 INFO:teuthology.orchestra.run.smithi139.stdout: debug rgw = 20 2021-09-06T11:23:05.852 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt require ssl = False 2021-09-06T11:23:05.852 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt s3 kms backend = testing 2021-09-06T11:23:05.852 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T11:23:05.852 INFO:teuthology.orchestra.run.smithi139.stdout: rgw curl low speed time = 300 2021-09-06T11:23:05.853 INFO:teuthology.orchestra.run.smithi139.stdout: rgw data log num shards = 4 2021-09-06T11:23:05.853 INFO:teuthology.orchestra.run.smithi139.stdout: rgw md log max shards = 4 2021-09-06T11:23:05.853 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync log trim interval = 0 2021-09-06T11:23:05.853 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync meta inject err probability = 0.1 2021-09-06T11:23:05.854 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync obj etag verify = True 2021-09-06T11:23:05.854 INFO:teuthology.orchestra.run.smithi139.stdout: setgroup = ceph 2021-09-06T11:23:05.854 INFO:teuthology.orchestra.run.smithi139.stdout: setuser = ceph 2021-09-06T11:23:05.854 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.a] 2021-09-06T11:23:05.854 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.0] 2021-09-06T11:23:05.855 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 10 2021-09-06T11:23:05.855 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.1] 2021-09-06T11:23:05.855 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 1000 2021-09-06T11:23:05.855 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.2] 2021-09-06T11:23:05.855 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 10000 2021-09-06T11:23:05.857 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-09-06T11:23:05.857 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-09-06T11:23:05.946 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-06T11:23:05.947 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:05.947 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-09-06T11:23:05.961 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:05.962 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-09-06T11:23:06.015 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2021-09-06T11:23:06.016 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:23:06.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/c2.keyring 2021-09-06T11:23:06.017 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-06T11:23:06.056 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-09-06T11:23:06.057 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-06T11:23:06.113 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi139.front.sepia.ceph.com 2021-09-06T11:23:06.114 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:06.114 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/c2.keyring 2021-09-06T11:23:06.114 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-06T11:23:06.147 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:06.148 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-06T11:23:06.201 INFO:tasks.ceph:Setting up mon nodes... 2021-09-06T11:23:06.202 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-06T11:23:06.202 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-09-06T11:23:06.286 INFO:teuthology.orchestra.run.smithi139.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-09-06T11:23:06.289 INFO:tasks.ceph:Setting up mds nodes... 2021-09-06T11:23:06.289 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-06T11:23:06.289 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-09-06T11:23:06.326 INFO:teuthology.orchestra.run.smithi139.stdout:creating /etc/ceph/c2.client.0.keyring 2021-09-06T11:23:06.337 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-09-06T11:23:06.414 INFO:teuthology.orchestra.run.smithi139.stdout:creating /etc/ceph/c2.client.1.keyring 2021-09-06T11:23:06.426 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-06T11:23:06.426 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi139.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-06T11:23:06.427 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-09-06T11:23:06.489 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T11:23:06.490 INFO:tasks.ceph:role: c2.osd.0 2021-09-06T11:23:06.490 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi139.front.sepia.ceph.com 2021-09-06T11:23:06.491 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-06T11:23:06.560 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T11:23:06.560 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T11:23:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T11:23:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 2021-09-06T11:23:06.561 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T11:23:06.561 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2021-09-06T11:23:06.562 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T11:23:06.562 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T11:23:06.562 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T11:23:06.562 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T11:23:06.921 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2021-09-06T11:23:06.923 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2021-09-06T11:23:06.923 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-09-06T11:23:07.061 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-09-06T11:23:07.089 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-09-06T11:23:07.153 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T11:23:07.154 INFO:tasks.ceph:role: c2.osd.1 2021-09-06T11:23:07.154 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi139.front.sepia.ceph.com 2021-09-06T11:23:07.154 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-06T11:23:07.224 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T11:23:07.225 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T11:23:07.225 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T11:23:07.225 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 2021-09-06T11:23:07.225 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T11:23:07.226 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2021-09-06T11:23:07.226 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T11:23:07.226 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T11:23:07.226 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T11:23:07.226 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T11:23:07.589 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2021-09-06T11:23:07.591 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2021-09-06T11:23:07.592 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-09-06T11:23:07.677 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-09-06T11:23:07.704 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-09-06T11:23:07.768 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T11:23:07.769 INFO:tasks.ceph:role: c2.osd.2 2021-09-06T11:23:07.769 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi139.front.sepia.ceph.com 2021-09-06T11:23:07.769 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-06T11:23:07.840 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T11:23:07.840 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T11:23:07.841 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T11:23:07.841 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 2021-09-06T11:23:07.841 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T11:23:07.841 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2021-09-06T11:23:07.841 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T11:23:07.842 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T11:23:07.842 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T11:23:07.842 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T11:23:08.205 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2021-09-06T11:23:08.206 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2021-09-06T11:23:08.207 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-09-06T11:23:08.314 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-09-06T11:23:08.343 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-06T11:23:08.430 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:08.431+0000 7fe7be3480c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-09-06T11:23:08.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:08.432+0000 7fe7be3480c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-09-06T11:23:08.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:08.432+0000 7fe7be3480c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-06T11:23:08.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:08.432+0000 7fe7be3480c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-06T11:23:08.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:08.432+0000 7fe7be3480c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-06T11:23:08.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:08.434+0000 7fe7be3480c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-09-06T11:23:10.521 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-09-06T11:23:10.552 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-06T11:23:10.638 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:10.637+0000 7fe82d9260c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-09-06T11:23:10.638 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:10.637+0000 7fe82d9260c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-09-06T11:23:10.638 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:10.637+0000 7fe82d9260c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-06T11:23:10.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:10.637+0000 7fe82d9260c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-06T11:23:10.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:10.637+0000 7fe82d9260c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-06T11:23:10.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:10.638+0000 7fe82d9260c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-09-06T11:23:12.722 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-09-06T11:23:12.750 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-06T11:23:12.834 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:12.834+0000 7f098b4b30c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-09-06T11:23:12.835 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:12.834+0000 7f098b4b30c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-09-06T11:23:12.835 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:12.834+0000 7f098b4b30c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-06T11:23:12.835 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:12.834+0000 7f098b4b30c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-06T11:23:12.836 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:12.834+0000 7f098b4b30c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-06T11:23:12.836 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:12.835+0000 7f098b4b30c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-09-06T11:23:14.924 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-09-06T11:23:14.992 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-06T11:23:14.992 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:14.993 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-09-06T11:23:15.058 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:15.058 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-09-06T11:23:15.123 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:15.124 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-09-06T11:23:15.188 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:15.188 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-09-06T11:23:15.251 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:15.251 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-09-06T11:23:15.306 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:15.306 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-09-06T11:23:15.359 INFO:tasks.ceph:Adding keys to all mons... 2021-09-06T11:23:15.360 DEBUG:teuthology.orchestra.run.smithi139:> sudo tee -a /etc/ceph/c2.keyring 2021-09-06T11:23:15.421 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr.x] 2021-09-06T11:23:15.421 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQAa+jVhPUUoERAAvHMpN7vX1J8liaJVZEfkGQ== 2021-09-06T11:23:15.421 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.0] 2021-09-06T11:23:15.422 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQAc+jVhtNa/GRAAK5GJoCnlXaeqxDHIYn1TSA== 2021-09-06T11:23:15.422 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.1] 2021-09-06T11:23:15.422 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQAe+jVhZhENJhAACTDn4iW4PUF71Ttmf0I2Kg== 2021-09-06T11:23:15.422 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.2] 2021-09-06T11:23:15.423 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQAg+jVhw2zKMRAAJTYtzsz6L0caNru13UFwpw== 2021-09-06T11:23:15.423 INFO:teuthology.orchestra.run.smithi139.stdout:[client.0] 2021-09-06T11:23:15.423 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQAa+jVhoVR/ExAAqX+h3A59YUeyK+b9RKAOeg== 2021-09-06T11:23:15.423 INFO:teuthology.orchestra.run.smithi139.stdout:[client.1] 2021-09-06T11:23:15.423 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQAa+jVh1xHAGBAA7WWz+h4PNi4deMtnllOhtQ== 2021-09-06T11:23:15.425 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-06T11:23:15.509 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T11:23:15.554 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T11:23:15.599 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T11:23:15.644 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-06T11:23:15.692 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-06T11:23:15.739 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-06T11:23:15.739 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-09-06T11:23:15.762 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-09-06T11:23:15.855 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-09-06T11:23:15.877 DEBUG:teuthology.orchestra.run.smithi139:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-09-06T11:23:15.931 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-09-06T11:23:15.932 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-09-06T11:23:15.932 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-09-06T11:23:15.973 INFO:tasks.ceph.c2.mon.a:Started 2021-09-06T11:23:15.974 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-09-06T11:23:15.974 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-09-06T11:23:15.975 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-09-06T11:23:15.977 INFO:tasks.ceph.c2.mgr.x:Started 2021-09-06T11:23:15.977 INFO:tasks.ceph:Setting crush tunables to default 2021-09-06T11:23:15.977 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd crush tunables default 2021-09-06T11:23:16.171 INFO:teuthology.orchestra.run.smithi139.stderr:adjusted tunables profile to default 2021-09-06T11:23:16.181 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-09-06T11:23:16.182 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:16.182 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-09-06T11:23:16.205 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:16.205 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-09-06T11:23:16.269 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-09-06T11:23:16.270 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-09-06T11:23:16.333 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd new 7a767443-a4f6-47b5-8063-6d793c771a50 0 2021-09-06T11:23:16.575 INFO:teuthology.orchestra.run.smithi139.stdout:0 2021-09-06T11:23:16.587 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd new 93618e21-a72c-45f0-9a3b-36ef21691424 1 2021-09-06T11:23:16.791 INFO:teuthology.orchestra.run.smithi139.stdout:1 2021-09-06T11:23:16.803 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd new 3bc7a3df-2b0c-48ef-b85c-2dd2cb01f449 2 2021-09-06T11:23:17.006 INFO:teuthology.orchestra.run.smithi139.stdout:2 2021-09-06T11:23:17.018 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-09-06T11:23:17.019 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-09-06T11:23:17.021 INFO:tasks.ceph.c2.osd.0:Started 2021-09-06T11:23:17.022 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-09-06T11:23:17.022 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-09-06T11:23:17.024 INFO:tasks.ceph.c2.osd.1:Started 2021-09-06T11:23:17.025 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-09-06T11:23:17.025 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-09-06T11:23:17.027 INFO:tasks.ceph.c2.osd.2:Started 2021-09-06T11:23:17.028 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-06T11:23:17.330 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-06T11:23:17.509 INFO:teuthology.misc.health.smithi139.stdout: 2021-09-06T11:23:17.510 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":5,"fsid":"4178eb40-61c1-44a5-9987-53bfc91286c0","created":"2021-09-06T11:23:16.077803+0000","modified":"2021-09-06T11:23:17.005787+0000","last_up_change":"0.000000","last_in_change":"2021-09-06T11:23:17.005787+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7a767443-a4f6-47b5-8063-6d793c771a50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"93618e21-a72c-45f0-9a3b-36ef21691424","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3bc7a3df-2b0c-48ef-b85c-2dd2cb01f449","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T11:23:17.520 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-06T11:23:17.616 INFO:tasks.ceph.c2.osd.0.smithi139.stderr:2021-09-06T11:23:17.616+0000 7fba9f0c60c0 -1 Falling back to public interface 2021-09-06T11:23:17.621 INFO:tasks.ceph.c2.osd.2.smithi139.stderr:2021-09-06T11:23:17.621+0000 7fa3e772e0c0 -1 Falling back to public interface 2021-09-06T11:23:17.621 INFO:tasks.ceph.c2.osd.1.smithi139.stderr:2021-09-06T11:23:17.621+0000 7f0558da10c0 -1 Falling back to public interface 2021-09-06T11:23:17.624 INFO:tasks.ceph.c2.osd.2.smithi139.stderr:2021-09-06T11:23:17.625+0000 7fa3e18f3700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6821/0: (98) Address already in use 2021-09-06T11:23:20.404 INFO:tasks.ceph.c2.osd.1.smithi139.stderr:2021-09-06T11:23:20.403+0000 7f0558da10c0 -1 osd.1 0 log_to_monitors {default=true} 2021-09-06T11:23:20.422 INFO:tasks.ceph.c2.osd.2.smithi139.stderr:2021-09-06T11:23:20.421+0000 7fa3e772e0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-09-06T11:23:20.423 INFO:tasks.ceph.c2.osd.0.smithi139.stderr:2021-09-06T11:23:20.421+0000 7fba9f0c60c0 -1 osd.0 0 log_to_monitors {default=true} 2021-09-06T11:23:22.636 INFO:tasks.ceph.c2.osd.1.smithi139.stderr:2021-09-06T11:23:22.635+0000 7f054f75f700 -1 osd.1 0 waiting for initial osdmap 2021-09-06T11:23:22.638 INFO:tasks.ceph.c2.osd.1.smithi139.stderr:2021-09-06T11:23:22.637+0000 7f05480f2700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T11:23:22.641 INFO:tasks.ceph.c2.osd.0.smithi139.stderr:2021-09-06T11:23:22.641+0000 7fba95a84700 -1 osd.0 0 waiting for initial osdmap 2021-09-06T11:23:22.643 INFO:tasks.ceph.c2.osd.0.smithi139.stderr:2021-09-06T11:23:22.642+0000 7fba8e417700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T11:23:23.136 INFO:tasks.ceph.c2.osd.2.smithi139.stderr:2021-09-06T11:23:23.135+0000 7fa3de0ec700 -1 osd.2 0 waiting for initial osdmap 2021-09-06T11:23:23.138 INFO:tasks.ceph.c2.osd.2.smithi139.stderr:2021-09-06T11:23:23.137+0000 7fa3d6a7f700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T11:23:23.824 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-06T11:23:24.105 INFO:teuthology.misc.health.smithi139.stdout: 2021-09-06T11:23:24.105 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":8,"fsid":"4178eb40-61c1-44a5-9987-53bfc91286c0","created":"2021-09-06T11:23:16.077803+0000","modified":"2021-09-06T11:23:23.640653+0000","last_up_change":"2021-09-06T11:23:23.640653+0000","last_in_change":"2021-09-06T11:23:17.005787+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7a767443-a4f6-47b5-8063-6d793c771a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":59589},{"type":"v1","addr":"172.21.15.139:6801","nonce":59589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":59589},{"type":"v1","addr":"172.21.15.139:6803","nonce":59589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":59589},{"type":"v1","addr":"172.21.15.139:6807","nonce":59589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":59589},{"type":"v1","addr":"172.21.15.139:6805","nonce":59589}]},"public_addr":"172.21.15.139:6801/59589","cluster_addr":"172.21.15.139:6803/59589","heartbeat_back_addr":"172.21.15.139:6807/59589","heartbeat_front_addr":"172.21.15.139:6805/59589","state":["exists","up"]},{"osd":1,"uuid":"93618e21-a72c-45f0-9a3b-36ef21691424","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":59590},{"type":"v1","addr":"172.21.15.139:6811","nonce":59590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":59590},{"type":"v1","addr":"172.21.15.139:6814","nonce":59590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":59590},{"type":"v1","addr":"172.21.15.139:6821","nonce":59590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":59590},{"type":"v1","addr":"172.21.15.139:6818","nonce":59590}]},"public_addr":"172.21.15.139:6811/59590","cluster_addr":"172.21.15.139:6814/59590","heartbeat_back_addr":"172.21.15.139:6821/59590","heartbeat_front_addr":"172.21.15.139:6818/59590","state":["exists","up"]},{"osd":2,"uuid":"3bc7a3df-2b0c-48ef-b85c-2dd2cb01f449","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":59591},{"type":"v1","addr":"172.21.15.139:6810","nonce":59591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":59591},{"type":"v1","addr":"172.21.15.139:6815","nonce":59591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":59591},{"type":"v1","addr":"172.21.15.139:6823","nonce":59591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":59591},{"type":"v1","addr":"172.21.15.139:6819","nonce":59591}]},"public_addr":"172.21.15.139:6810/59591","cluster_addr":"172.21.15.139:6815/59591","heartbeat_back_addr":"172.21.15.139:6823/59591","heartbeat_front_addr":"172.21.15.139:6819/59591","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T11:23:24.117 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-06T11:23:24.117 INFO:tasks.ceph:Creating RBD pool 2021-09-06T11:23:24.118 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd pool create rbd 8 2021-09-06T11:23:25.032 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'rbd' created 2021-09-06T11:23:25.045 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-06T11:23:26.037 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-06T11:23:26.051 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-09-06T11:23:26.051 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-06T11:23:26.052 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-06T11:23:26.370 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:26.371 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":10,"fsid":"4178eb40-61c1-44a5-9987-53bfc91286c0","created":"2021-09-06T11:23:16.077803+0000","modified":"2021-09-06T11:23:26.026031+0000","last_up_change":"2021-09-06T11:23:23.640653+0000","last_in_change":"2021-09-06T11:23:17.005787+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-06T11:23:24.415381+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-06T11:23:24.995884+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7a767443-a4f6-47b5-8063-6d793c771a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":59589},{"type":"v1","addr":"172.21.15.139:6801","nonce":59589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":59589},{"type":"v1","addr":"172.21.15.139:6803","nonce":59589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":59589},{"type":"v1","addr":"172.21.15.139:6807","nonce":59589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":59589},{"type":"v1","addr":"172.21.15.139:6805","nonce":59589}]},"public_addr":"172.21.15.139:6801/59589","cluster_addr":"172.21.15.139:6803/59589","heartbeat_back_addr":"172.21.15.139:6807/59589","heartbeat_front_addr":"172.21.15.139:6805/59589","state":["exists","up"]},{"osd":1,"uuid":"93618e21-a72c-45f0-9a3b-36ef21691424","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":59590},{"type":"v1","addr":"172.21.15.139:6811","nonce":59590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":59590},{"type":"v1","addr":"172.21.15.139:6814","nonce":59590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":59590},{"type":"v1","addr":"172.21.15.139:6821","nonce":59590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":59590},{"type":"v1","addr":"172.21.15.139:6818","nonce":59590}]},"public_addr":"172.21.15.139:6811/59590","cluster_addr":"172.21.15.139:6814/59590","heartbeat_back_addr":"172.21.15.139:6821/59590","heartbeat_front_addr":"172.21.15.139:6818/59590","state":["exists","up"]},{"osd":2,"uuid":"3bc7a3df-2b0c-48ef-b85c-2dd2cb01f449","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":59591},{"type":"v1","addr":"172.21.15.139:6810","nonce":59591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":59591},{"type":"v1","addr":"172.21.15.139:6815","nonce":59591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":59591},{"type":"v1","addr":"172.21.15.139:6823","nonce":59591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":59591},{"type":"v1","addr":"172.21.15.139:6819","nonce":59591}]},"public_addr":"172.21.15.139:6810/59591","cluster_addr":"172.21.15.139:6815/59591","heartbeat_back_addr":"172.21.15.139:6823/59591","heartbeat_front_addr":"172.21.15.139:6819/59591","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:23:21.408217+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:23:21.424682+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:23:21.407724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T11:23:26.384 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-09-06T11:23:24.415381+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2021-09-06T11:23:24.995884+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2021-09-06T11:23:26.385 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-09-06T11:23:26.676 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 8 2021-09-06T11:23:26.689 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get .mgr pg_num 2021-09-06T11:23:26.973 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 1 2021-09-06T11:23:26.986 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-09-06T11:23:26.986 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-09-06T11:23:26.987 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-09-06T11:23:27.312 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:27.324 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":59272},{"type":"v1","addr":"172.21.15.139:6825","nonce":59272}]},"active_addr":"172.21.15.139:6825/59272","active_change":"2021-09-06T11:23:20.961513+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":1309620149}]},{"addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":173969543}]},{"addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":3254986323}]},{"addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":1296115508}]}]}} 2021-09-06T11:23:27.326 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-09-06T11:23:27.326 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-09-06T11:23:27.327 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-06T11:23:27.607 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:27.608 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":11,"fsid":"4178eb40-61c1-44a5-9987-53bfc91286c0","created":"2021-09-06T11:23:16.077803+0000","modified":"2021-09-06T11:23:27.030383+0000","last_up_change":"2021-09-06T11:23:23.640653+0000","last_in_change":"2021-09-06T11:23:17.005787+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-06T11:23:24.415381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-06T11:23:24.995884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7a767443-a4f6-47b5-8063-6d793c771a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":59589},{"type":"v1","addr":"172.21.15.139:6801","nonce":59589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":59589},{"type":"v1","addr":"172.21.15.139:6803","nonce":59589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":59589},{"type":"v1","addr":"172.21.15.139:6807","nonce":59589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":59589},{"type":"v1","addr":"172.21.15.139:6805","nonce":59589}]},"public_addr":"172.21.15.139:6801/59589","cluster_addr":"172.21.15.139:6803/59589","heartbeat_back_addr":"172.21.15.139:6807/59589","heartbeat_front_addr":"172.21.15.139:6805/59589","state":["exists","up"]},{"osd":1,"uuid":"93618e21-a72c-45f0-9a3b-36ef21691424","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":59590},{"type":"v1","addr":"172.21.15.139:6811","nonce":59590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":59590},{"type":"v1","addr":"172.21.15.139:6814","nonce":59590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":59590},{"type":"v1","addr":"172.21.15.139:6821","nonce":59590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":59590},{"type":"v1","addr":"172.21.15.139:6818","nonce":59590}]},"public_addr":"172.21.15.139:6811/59590","cluster_addr":"172.21.15.139:6814/59590","heartbeat_back_addr":"172.21.15.139:6821/59590","heartbeat_front_addr":"172.21.15.139:6818/59590","state":["exists","up"]},{"osd":2,"uuid":"3bc7a3df-2b0c-48ef-b85c-2dd2cb01f449","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":59591},{"type":"v1","addr":"172.21.15.139:6810","nonce":59591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":59591},{"type":"v1","addr":"172.21.15.139:6815","nonce":59591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":59591},{"type":"v1","addr":"172.21.15.139:6823","nonce":59591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":59591},{"type":"v1","addr":"172.21.15.139:6819","nonce":59591}]},"public_addr":"172.21.15.139:6810/59591","cluster_addr":"172.21.15.139:6815/59591","heartbeat_back_addr":"172.21.15.139:6823/59591","heartbeat_front_addr":"172.21.15.139:6819/59591","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:23:21.408217+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:23:21.424682+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:23:21.407724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T11:23:27.620 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-09-06T11:23:27.621 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-06T11:23:27.901 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:27.901 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":11,"fsid":"4178eb40-61c1-44a5-9987-53bfc91286c0","created":"2021-09-06T11:23:16.077803+0000","modified":"2021-09-06T11:23:27.030383+0000","last_up_change":"2021-09-06T11:23:23.640653+0000","last_in_change":"2021-09-06T11:23:17.005787+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-06T11:23:24.415381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-06T11:23:24.995884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7a767443-a4f6-47b5-8063-6d793c771a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":59589},{"type":"v1","addr":"172.21.15.139:6801","nonce":59589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":59589},{"type":"v1","addr":"172.21.15.139:6803","nonce":59589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":59589},{"type":"v1","addr":"172.21.15.139:6807","nonce":59589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":59589},{"type":"v1","addr":"172.21.15.139:6805","nonce":59589}]},"public_addr":"172.21.15.139:6801/59589","cluster_addr":"172.21.15.139:6803/59589","heartbeat_back_addr":"172.21.15.139:6807/59589","heartbeat_front_addr":"172.21.15.139:6805/59589","state":["exists","up"]},{"osd":1,"uuid":"93618e21-a72c-45f0-9a3b-36ef21691424","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":59590},{"type":"v1","addr":"172.21.15.139:6811","nonce":59590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":59590},{"type":"v1","addr":"172.21.15.139:6814","nonce":59590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":59590},{"type":"v1","addr":"172.21.15.139:6821","nonce":59590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":59590},{"type":"v1","addr":"172.21.15.139:6818","nonce":59590}]},"public_addr":"172.21.15.139:6811/59590","cluster_addr":"172.21.15.139:6814/59590","heartbeat_back_addr":"172.21.15.139:6821/59590","heartbeat_front_addr":"172.21.15.139:6818/59590","state":["exists","up"]},{"osd":2,"uuid":"3bc7a3df-2b0c-48ef-b85c-2dd2cb01f449","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":59591},{"type":"v1","addr":"172.21.15.139:6810","nonce":59591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":59591},{"type":"v1","addr":"172.21.15.139:6815","nonce":59591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":59591},{"type":"v1","addr":"172.21.15.139:6823","nonce":59591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":59591},{"type":"v1","addr":"172.21.15.139:6819","nonce":59591}]},"public_addr":"172.21.15.139:6810/59591","cluster_addr":"172.21.15.139:6815/59591","heartbeat_back_addr":"172.21.15.139:6823/59591","heartbeat_front_addr":"172.21.15.139:6819/59591","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:23:21.408217+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:23:21.424682+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T11:23:21.407724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T11:23:27.914 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-09-06T11:23:28.058 INFO:teuthology.orchestra.run.smithi139.stdout:34359738370 2021-09-06T11:23:28.059 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-09-06T11:23:28.203 INFO:teuthology.orchestra.run.smithi139.stdout:34359738372 2021-09-06T11:23:28.203 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-09-06T11:23:28.347 INFO:teuthology.orchestra.run.smithi139.stdout:34359738371 2021-09-06T11:23:28.347 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-06T11:23:28.628 INFO:teuthology.orchestra.run.smithi139.stdout:34359738369 2021-09-06T11:23:28.641 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2021-09-06T11:23:29.643 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-06T11:23:29.938 INFO:teuthology.orchestra.run.smithi139.stdout:34359738370 2021-09-06T11:23:29.951 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2021-09-06T11:23:29.952 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-09-06T11:23:30.233 INFO:teuthology.orchestra.run.smithi139.stdout:34359738372 2021-09-06T11:23:30.246 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2021-09-06T11:23:30.246 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-09-06T11:23:30.528 INFO:teuthology.orchestra.run.smithi139.stdout:34359738371 2021-09-06T11:23:30.542 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2021-09-06T11:23:30.542 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-09-06T11:23:30.543 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-06T11:23:30.817 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:30.818 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2021-09-06T11:23:30.829 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2021-09-06T11:23:28.968777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24636,"kb_used_data":1444,"kb_used_omap":0,"kb_used_meta":23168,"kb_avail":314548164,"statfs":{"total":322122547200,"available":322097319936,"internally_reserved":0,"allocated":1478656,"data_stored":1258225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23724032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.038334+0000","last_change":"2021-09-06T11:23:26.037148+0000","last_active":"2021-09-06T11:23:26.038334+0000","last_peered":"2021-09-06T11:23:26.038334+0000","last_clean":"2021-09-06T11:23:26.038334+0000","last_became_active":"2021-09-06T11:23:26.036891+0000","last_became_peered":"2021-09-06T11:23:26.036891+0000","last_unstale":"2021-09-06T11:23:26.038334+0000","last_undegraded":"2021-09-06T11:23:26.038334+0000","last_fullsized":"2021-09-06T11:23:26.038334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.038805+0000","last_change":"2021-09-06T11:23:26.037791+0000","last_active":"2021-09-06T11:23:26.038805+0000","last_peered":"2021-09-06T11:23:26.038805+0000","last_clean":"2021-09-06T11:23:26.038805+0000","last_became_active":"2021-09-06T11:23:26.035333+0000","last_became_peered":"2021-09-06T11:23:26.035333+0000","last_unstale":"2021-09-06T11:23:26.038805+0000","last_undegraded":"2021-09-06T11:23:26.038805+0000","last_fullsized":"2021-09-06T11:23:26.038805+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.038539+0000","last_change":"2021-09-06T11:23:26.034520+0000","last_active":"2021-09-06T11:23:26.038539+0000","last_peered":"2021-09-06T11:23:26.038539+0000","last_clean":"2021-09-06T11:23:26.038539+0000","last_became_active":"2021-09-06T11:23:26.034263+0000","last_became_peered":"2021-09-06T11:23:26.034263+0000","last_unstale":"2021-09-06T11:23:26.038539+0000","last_undegraded":"2021-09-06T11:23:26.038539+0000","last_fullsized":"2021-09-06T11:23:26.038539+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.036387+0000","last_change":"2021-09-06T11:23:26.034389+0000","last_active":"2021-09-06T11:23:26.036387+0000","last_peered":"2021-09-06T11:23:26.036387+0000","last_clean":"2021-09-06T11:23:26.036387+0000","last_became_active":"2021-09-06T11:23:26.034086+0000","last_became_peered":"2021-09-06T11:23:26.034086+0000","last_unstale":"2021-09-06T11:23:26.036387+0000","last_undegraded":"2021-09-06T11:23:26.036387+0000","last_fullsized":"2021-09-06T11:23:26.036387+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.037464+0000","last_change":"2021-09-06T11:23:26.036111+0000","last_active":"2021-09-06T11:23:26.037464+0000","last_peered":"2021-09-06T11:23:26.037464+0000","last_clean":"2021-09-06T11:23:26.037464+0000","last_became_active":"2021-09-06T11:23:26.035784+0000","last_became_peered":"2021-09-06T11:23:26.035784+0000","last_unstale":"2021-09-06T11:23:26.037464+0000","last_undegraded":"2021-09-06T11:23:26.037464+0000","last_fullsized":"2021-09-06T11:23:26.037464+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.518582+0000","last_change":"2021-09-06T11:23:26.038234+0000","last_active":"2021-09-06T11:23:26.518582+0000","last_peered":"2021-09-06T11:23:26.518582+0000","last_clean":"2021-09-06T11:23:26.518582+0000","last_became_active":"2021-09-06T11:23:26.037935+0000","last_became_peered":"2021-09-06T11:23:26.037935+0000","last_unstale":"2021-09-06T11:23:26.518582+0000","last_undegraded":"2021-09-06T11:23:26.518582+0000","last_fullsized":"2021-09-06T11:23:26.518582+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.038775+0000","last_change":"2021-09-06T11:23:26.037425+0000","last_active":"2021-09-06T11:23:26.038775+0000","last_peered":"2021-09-06T11:23:26.038775+0000","last_clean":"2021-09-06T11:23:26.038775+0000","last_became_active":"2021-09-06T11:23:26.037037+0000","last_became_peered":"2021-09-06T11:23:26.037037+0000","last_unstale":"2021-09-06T11:23:26.038775+0000","last_undegraded":"2021-09-06T11:23:26.038775+0000","last_fullsized":"2021-09-06T11:23:26.038775+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.037242+0000","last_change":"2021-09-06T11:23:26.036249+0000","last_active":"2021-09-06T11:23:26.037242+0000","last_peered":"2021-09-06T11:23:26.037242+0000","last_clean":"2021-09-06T11:23:26.037242+0000","last_became_active":"2021-09-06T11:23:26.035536+0000","last_became_peered":"2021-09-06T11:23:26.035536+0000","last_unstale":"2021-09-06T11:23:26.037242+0000","last_undegraded":"2021-09-06T11:23:26.037242+0000","last_fullsized":"2021-09-06T11:23:26.037242+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.038843+0000","last_change":"2021-09-06T11:23:26.035581+0000","last_active":"2021-09-06T11:23:26.038843+0000","last_peered":"2021-09-06T11:23:26.038843+0000","last_clean":"2021-09-06T11:23:26.038843+0000","last_became_active":"2021-09-06T11:23:26.035326+0000","last_became_peered":"2021-09-06T11:23:26.035326+0000","last_unstale":"2021-09-06T11:23:26.038843+0000","last_undegraded":"2021-09-06T11:23:26.038843+0000","last_fullsized":"2021-09-06T11:23:26.038843+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8492,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104849108,"statfs":{"total":107374182400,"available":107365486592,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7588,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850012,"statfs":{"total":107374182400,"available":107366412288,"internally_reserved":0,"allocated":94208,"data_stored":25105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8556,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849044,"statfs":{"total":107374182400,"available":107365421056,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-06T11:23:30.830 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-06T11:23:31.108 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:31.110 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2021-09-06T11:23:31.121 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-09-06T11:23:30.969190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24636,"kb_used_data":1444,"kb_used_omap":0,"kb_used_meta":23168,"kb_avail":314548164,"statfs":{"total":322122547200,"available":322097319936,"internally_reserved":0,"allocated":1478656,"data_stored":1258225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23724032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000413"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.038334+0000","last_change":"2021-09-06T11:23:26.037148+0000","last_active":"2021-09-06T11:23:26.038334+0000","last_peered":"2021-09-06T11:23:26.038334+0000","last_clean":"2021-09-06T11:23:26.038334+0000","last_became_active":"2021-09-06T11:23:26.036891+0000","last_became_peered":"2021-09-06T11:23:26.036891+0000","last_unstale":"2021-09-06T11:23:26.038334+0000","last_undegraded":"2021-09-06T11:23:26.038334+0000","last_fullsized":"2021-09-06T11:23:26.038334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.038805+0000","last_change":"2021-09-06T11:23:26.037791+0000","last_active":"2021-09-06T11:23:26.038805+0000","last_peered":"2021-09-06T11:23:26.038805+0000","last_clean":"2021-09-06T11:23:26.038805+0000","last_became_active":"2021-09-06T11:23:26.035333+0000","last_became_peered":"2021-09-06T11:23:26.035333+0000","last_unstale":"2021-09-06T11:23:26.038805+0000","last_undegraded":"2021-09-06T11:23:26.038805+0000","last_fullsized":"2021-09-06T11:23:26.038805+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.038539+0000","last_change":"2021-09-06T11:23:26.034520+0000","last_active":"2021-09-06T11:23:26.038539+0000","last_peered":"2021-09-06T11:23:26.038539+0000","last_clean":"2021-09-06T11:23:26.038539+0000","last_became_active":"2021-09-06T11:23:26.034263+0000","last_became_peered":"2021-09-06T11:23:26.034263+0000","last_unstale":"2021-09-06T11:23:26.038539+0000","last_undegraded":"2021-09-06T11:23:26.038539+0000","last_fullsized":"2021-09-06T11:23:26.038539+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.036387+0000","last_change":"2021-09-06T11:23:26.034389+0000","last_active":"2021-09-06T11:23:26.036387+0000","last_peered":"2021-09-06T11:23:26.036387+0000","last_clean":"2021-09-06T11:23:26.036387+0000","last_became_active":"2021-09-06T11:23:26.034086+0000","last_became_peered":"2021-09-06T11:23:26.034086+0000","last_unstale":"2021-09-06T11:23:26.036387+0000","last_undegraded":"2021-09-06T11:23:26.036387+0000","last_fullsized":"2021-09-06T11:23:26.036387+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.037464+0000","last_change":"2021-09-06T11:23:26.036111+0000","last_active":"2021-09-06T11:23:26.037464+0000","last_peered":"2021-09-06T11:23:26.037464+0000","last_clean":"2021-09-06T11:23:26.037464+0000","last_became_active":"2021-09-06T11:23:26.035784+0000","last_became_peered":"2021-09-06T11:23:26.035784+0000","last_unstale":"2021-09-06T11:23:26.037464+0000","last_undegraded":"2021-09-06T11:23:26.037464+0000","last_fullsized":"2021-09-06T11:23:26.037464+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.518582+0000","last_change":"2021-09-06T11:23:26.038234+0000","last_active":"2021-09-06T11:23:26.518582+0000","last_peered":"2021-09-06T11:23:26.518582+0000","last_clean":"2021-09-06T11:23:26.518582+0000","last_became_active":"2021-09-06T11:23:26.037935+0000","last_became_peered":"2021-09-06T11:23:26.037935+0000","last_unstale":"2021-09-06T11:23:26.518582+0000","last_undegraded":"2021-09-06T11:23:26.518582+0000","last_fullsized":"2021-09-06T11:23:26.518582+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.038775+0000","last_change":"2021-09-06T11:23:26.037425+0000","last_active":"2021-09-06T11:23:26.038775+0000","last_peered":"2021-09-06T11:23:26.038775+0000","last_clean":"2021-09-06T11:23:26.038775+0000","last_became_active":"2021-09-06T11:23:26.037037+0000","last_became_peered":"2021-09-06T11:23:26.037037+0000","last_unstale":"2021-09-06T11:23:26.038775+0000","last_undegraded":"2021-09-06T11:23:26.038775+0000","last_fullsized":"2021-09-06T11:23:26.038775+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.037242+0000","last_change":"2021-09-06T11:23:26.036249+0000","last_active":"2021-09-06T11:23:26.037242+0000","last_peered":"2021-09-06T11:23:26.037242+0000","last_clean":"2021-09-06T11:23:26.037242+0000","last_became_active":"2021-09-06T11:23:26.035536+0000","last_became_peered":"2021-09-06T11:23:26.035536+0000","last_unstale":"2021-09-06T11:23:26.037242+0000","last_undegraded":"2021-09-06T11:23:26.037242+0000","last_fullsized":"2021-09-06T11:23:26.037242+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-06T11:23:26.038843+0000","last_change":"2021-09-06T11:23:26.035581+0000","last_active":"2021-09-06T11:23:26.038843+0000","last_peered":"2021-09-06T11:23:26.038843+0000","last_clean":"2021-09-06T11:23:26.038843+0000","last_became_active":"2021-09-06T11:23:26.035326+0000","last_became_peered":"2021-09-06T11:23:26.035326+0000","last_unstale":"2021-09-06T11:23:26.038843+0000","last_undegraded":"2021-09-06T11:23:26.038843+0000","last_fullsized":"2021-09-06T11:23:26.038843+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T11:23:25.022919+0000","last_clean_scrub_stamp":"2021-09-06T11:23:25.022919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8492,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104849108,"statfs":{"total":107374182400,"available":107365486592,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7588,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850012,"statfs":{"total":107374182400,"available":107366412288,"internally_reserved":0,"allocated":94208,"data_stored":25105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8556,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849044,"statfs":{"total":107374182400,"available":107365421056,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-06T11:23:31.122 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-09-06T11:23:31.122 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-09-06T11:23:31.123 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-09-06T11:23:31.123 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-09-06T11:23:31.464 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-09-06T11:23:31.464 INFO:teuthology.orchestra.run.smithi139.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-06T11:23:31.477 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-09-06T11:23:31.477 INFO:teuthology.run_tasks:Running task rgw... 2021-09-06T11:23:31.491 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-09-06T11:23:31.491 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-09-06T11:23:31.492 INFO:tasks.rgw:Creating data pools 2021-09-06T11:23:31.492 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-09-06T11:23:31.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-06T11:23:32.510 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'default.rgw.buckets.data' created 2021-09-06T11:23:32.522 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-06T11:23:33.505 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-06T11:23:33.522 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-06T11:23:34.508 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'default.rgw.buckets.index' created 2021-09-06T11:23:34.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-06T11:23:35.511 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-06T11:23:35.530 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-09-06T11:23:35.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-06T11:23:35.823 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-06T11:23:35.833 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-06T11:23:36.506 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-06T11:23:36.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-06T11:23:36.798 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-06T11:23:36.808 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-06T11:23:37.508 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-06T11:23:37.519 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-09-06T11:23:37.519 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-06T11:23:38.072 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'default.rgw.buckets.data' created 2021-09-06T11:23:38.087 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-06T11:23:39.098 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-06T11:23:39.117 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-06T11:23:40.078 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'default.rgw.buckets.index' created 2021-09-06T11:23:40.095 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-06T11:23:41.095 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-06T11:23:41.117 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-09-06T11:23:41.118 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-06T11:23:41.400 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-06T11:23:41.411 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-06T11:23:42.093 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-06T11:23:42.105 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-06T11:23:42.384 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-06T11:23:42.394 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-06T11:23:43.098 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-06T11:23:43.109 DEBUG:tasks.rgw:Pools created 2021-09-06T11:23:43.110 INFO:tasks.rgw:Configuring compression type = random 2021-09-06T11:23:43.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-09-06T11:23:43.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-09-06T11:23:43.111 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-09-06T11:23:43.213 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:23:43.214 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:23:43.220 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:43.220+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.539 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.539+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.539 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.539+0000 7ff37c4b0380 20 realm 2021-09-06T11:23:44.539 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.539+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.540 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.539+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.541 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.539+0000 7ff37c4b0380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-06T11:23:44.541 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.539+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.541 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.540+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.541 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.540+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.542 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.541+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.543 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.541+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.543 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.541+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.543 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.541+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.543 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.542+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.544 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.542+0000 7ff37c4b0380 10 Creating default zonegroup 2021-09-06T11:23:44.544 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.542+0000 7ff37c4b0380 20 get_system_obj_state: rctx=0x7fff076c53e0 obj=:.pools.avail state=0x55ec2522dd70 s->prefetch_data=0 2021-09-06T11:23:44.544 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.543+0000 7ff37c4b0380 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-06T11:23:44.556 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.557+0000 7ff37c4b0380 10 WARNING: store->list_zones() returned r=-2 2021-09-06T11:23:44.557 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.557+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.557 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.557+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.560 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.560+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.561 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.560+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.561 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.560+0000 7ff37c4b0380 10 could not read realm id: (2) No such file or directory 2021-09-06T11:23:44.561 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.560+0000 7ff37c4b0380 10 WARNING: failed to set zone as default, r=-22 2021-09-06T11:23:44.561 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.560+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.562 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.561+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.563 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.564+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.565 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.564+0000 7ff37c4b0380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T11:23:44.565 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.564+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.565 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.564+0000 7ff37c4b0380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:23:44.566 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.564+0000 7ff37c4b0380 10 Using default name default 2021-09-06T11:23:44.566 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.564+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.566 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.565+0000 7ff37c4b0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:44.567 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.565+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.567 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.565+0000 7ff37c4b0380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T11:23:44.567 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.565+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.568 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.565+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.568 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.565+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.568 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.566+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.569 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.566+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.569 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.566+0000 7ff37c4b0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:44.577 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.577+0000 7ff37c4b0380 20 RGWRados::pool_iterate: got zone_info.99ed0e5e-8448-467f-bbfd-3849514b595a 2021-09-06T11:23:44.577 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.577+0000 7ff37c4b0380 20 RGWRados::pool_iterate: got zonegroup_info.eb837787-11a2-48e7-9fd9-f26232451aaa 2021-09-06T11:23:44.577 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.577+0000 7ff37c4b0380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T11:23:44.578 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.577+0000 7ff37c4b0380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T11:23:44.578 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.577+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.578 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.577+0000 7ff37c4b0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:44.578 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.577+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.580 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.577+0000 7ff37c4b0380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T11:23:44.580 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.577+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.580 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.578+0000 7ff37c4b0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:44.581 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.578+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.581 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.578+0000 7ff37c4b0380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:23:44.581 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.578+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.581 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.579+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.581 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.579+0000 7ff37c4b0380 20 zone default found 2021-09-06T11:23:44.582 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.579+0000 7ff37c4b0380 1 Realm: () 2021-09-06T11:23:44.582 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.579+0000 7ff37c4b0380 1 ZoneGroup: default (eb837787-11a2-48e7-9fd9-f26232451aaa) 2021-09-06T11:23:44.582 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.579+0000 7ff37c4b0380 1 Zone: default (99ed0e5e-8448-467f-bbfd-3849514b595a) 2021-09-06T11:23:44.582 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.579+0000 7ff37c4b0380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-06T11:23:44.583 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.579+0000 7ff37c4b0380 20 zonegroup default 2021-09-06T11:23:44.583 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.579+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:44.583 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.579+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:44.583 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:44.579+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:46.543 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:46.543+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:46.544 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:46.543+0000 7ff37c4b0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:46.545 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:46.545+0000 7ff37c4b0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:46.545 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:46.545+0000 7ff37c4b0380 20 started sync module instance, tier type = 2021-09-06T11:23:46.545 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:46.545+0000 7ff37c4b0380 20 started zone id=99ed0e5e-8448-467f-bbfd-3849514b595a (name=default) with tier type = 2021-09-06T11:23:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.570+0000 7ff37c4b0380 20 add_watcher() i=0 2021-09-06T11:23:48.571 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.570+0000 7ff37c4b0380 20 add_watcher() i=1 2021-09-06T11:23:48.571 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.570+0000 7ff37c4b0380 20 add_watcher() i=2 2021-09-06T11:23:48.571 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.570+0000 7ff37c4b0380 20 add_watcher() i=3 2021-09-06T11:23:48.572 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.570+0000 7ff37c4b0380 20 add_watcher() i=4 2021-09-06T11:23:48.572 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.570+0000 7ff37c4b0380 20 add_watcher() i=5 2021-09-06T11:23:48.572 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.570+0000 7ff37c4b0380 20 add_watcher() i=6 2021-09-06T11:23:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.571+0000 7ff37c4b0380 20 add_watcher() i=7 2021-09-06T11:23:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.571+0000 7ff37c4b0380 2 all 8 watchers are set, enabling cache 2021-09-06T11:23:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.572+0000 7ff37c4b0380 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-06T11:23:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.572+0000 7ff37c4b0380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-06T11:23:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.572+0000 7ff37c4b0380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T11:23:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.572+0000 7ff37c4b0380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-06T11:23:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.572+0000 7ff37c4b0380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-06T11:23:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.572+0000 7ff37c4b0380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T11:23:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.572+0000 7ff37c4b0380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-06T11:23:48.576 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.572+0000 7ff37c4b0380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-06T11:23:48.576 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.572+0000 7ff37c4b0380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T11:23:48.576 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.573+0000 7ff37c4b0380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-06T11:23:48.576 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.573+0000 7ff37c4b0380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-06T11:23:48.577 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.573+0000 7ff37c4b0380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T11:23:48.577 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.573+0000 7ff37c4b0380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-06T11:23:48.577 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.573+0000 7ff37c4b0380 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-06T11:23:48.578 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.575+0000 7ff37c4b0380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T11:23:48.578 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.578+0000 7ff340fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T11:23:48.579 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.578+0000 7ff37c4b0380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T11:23:48.579 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:48.578+0000 7ff37c4b0380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T11:23:52.548 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.548+0000 7ff37c4b0380 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-06T11:23:52.549 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.548+0000 7ff37c4b0380 5 note: GC not initialized 2021-09-06T11:23:52.550 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.550+0000 7ff2a37fe700 20 reqs_thread_entry: start 2021-09-06T11:23:52.550 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.550+0000 7ff2a17fa700 20 reqs_thread_entry: start 2021-09-06T11:23:52.551 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.550+0000 7ff37c4b0380 20 init_complete bucket index max shards: 11 2021-09-06T11:23:52.551 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.550+0000 7ff37c4b0380 10 rgw notify: Started notification manager with: 1 workers 2021-09-06T11:23:52.551 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.550+0000 7ff28f7fe700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 6 11:24:22 2021 2021-09-06T11:23:52.551 INFO:teuthology.orchestra.run.smithi094.stderr: 2021-09-06T11:23:52.552 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.550+0000 7ff28effd700 20 reqs_thread_entry: start 2021-09-06T11:23:52.573 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-09-06T11:23:52.573 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.572+0000 7ff37c4b0380 20 remove_watcher() i=0 2021-09-06T11:23:52.573 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.572+0000 7ff37c4b0380 2 removed watcher, disabling cache 2021-09-06T11:23:52.574 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.574+0000 7ff37c4b0380 20 remove_watcher() i=1 2021-09-06T11:23:52.575 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.575+0000 7ff37c4b0380 20 remove_watcher() i=2 2021-09-06T11:23:52.576 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.576+0000 7ff37c4b0380 20 remove_watcher() i=3 2021-09-06T11:23:52.578 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.578+0000 7ff37c4b0380 20 remove_watcher() i=4 2021-09-06T11:23:52.579 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.579+0000 7ff37c4b0380 20 remove_watcher() i=5 2021-09-06T11:23:52.580 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.580+0000 7ff37c4b0380 20 remove_watcher() i=6 2021-09-06T11:23:52.582 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.582+0000 7ff37c4b0380 20 remove_watcher() i=7 2021-09-06T11:23:52.582 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.583+0000 7ff340fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T11:23:52.588 DEBUG:tasks.util.rgw: json result: [] 2021-09-06T11:23:52.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T11:23:52.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T11:23:52.589 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-06T11:23:52.634 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:23:52.634 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:23:52.639 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.639+0000 7f3017cbe380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T11:23:52.640 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.639+0000 7f3017cbe380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T11:23:52.643 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.643+0000 7f2f4ffff700 20 reqs_thread_entry: start 2021-09-06T11:23:52.643 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.643+0000 7f3017cbe380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-06T11:23:52.643 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.643+0000 7f3017cbe380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.644 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.644+0000 7f3017cbe380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:52.644 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.644+0000 7f3017cbe380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.645 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.645+0000 7f3017cbe380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T11:23:52.656 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.656+0000 7f3017cbe380 20 RGWRados::pool_iterate: got zone_info.99ed0e5e-8448-467f-bbfd-3849514b595a 2021-09-06T11:23:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.656+0000 7f3017cbe380 20 RGWRados::pool_iterate: got zonegroup_info.eb837787-11a2-48e7-9fd9-f26232451aaa 2021-09-06T11:23:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.656+0000 7f3017cbe380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T11:23:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.656+0000 7f3017cbe380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T11:23:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.656+0000 7f3017cbe380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.657 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.656+0000 7f3017cbe380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T11:23:52.658 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.656+0000 7f3017cbe380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.658 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.656+0000 7f3017cbe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.658 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.656+0000 7f3017cbe380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.659 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.657+0000 7f3017cbe380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:52.659 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.657+0000 7f3017cbe380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.659 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.657+0000 7f3017cbe380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:23:52.661 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:23:52.661 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "99ed0e5e-8448-467f-bbfd-3849514b595a", 2021-09-06T11:23:52.661 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default", 2021-09-06T11:23:52.661 INFO:teuthology.orchestra.run.smithi094.stdout: "domain_root": "default.rgw.meta:root", 2021-09-06T11:23:52.662 INFO:teuthology.orchestra.run.smithi094.stdout: "control_pool": "default.rgw.control", 2021-09-06T11:23:52.662 INFO:teuthology.orchestra.run.smithi094.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-06T11:23:52.662 INFO:teuthology.orchestra.run.smithi094.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-06T11:23:52.662 INFO:teuthology.orchestra.run.smithi094.stdout: "log_pool": "default.rgw.log", 2021-09-06T11:23:52.663 INFO:teuthology.orchestra.run.smithi094.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-06T11:23:52.663 INFO:teuthology.orchestra.run.smithi094.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-06T11:23:52.663 INFO:teuthology.orchestra.run.smithi094.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-06T11:23:52.663 INFO:teuthology.orchestra.run.smithi094.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-06T11:23:52.663 INFO:teuthology.orchestra.run.smithi094.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-06T11:23:52.664 INFO:teuthology.orchestra.run.smithi094.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-06T11:23:52.664 INFO:teuthology.orchestra.run.smithi094.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-06T11:23:52.664 INFO:teuthology.orchestra.run.smithi094.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-06T11:23:52.664 INFO:teuthology.orchestra.run.smithi094.stdout: "otp_pool": "default.rgw.otp", 2021-09-06T11:23:52.665 INFO:teuthology.orchestra.run.smithi094.stdout: "system_key": { 2021-09-06T11:23:52.665 INFO:teuthology.orchestra.run.smithi094.stdout: "access_key": "", 2021-09-06T11:23:52.665 INFO:teuthology.orchestra.run.smithi094.stdout: "secret_key": "" 2021-09-06T11:23:52.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:23:52.665 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_pools": [ 2021-09-06T11:23:52.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:23:52.666 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "default-placement", 2021-09-06T11:23:52.666 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:23:52.666 INFO:teuthology.orchestra.run.smithi094.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-06T11:23:52.666 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": { 2021-09-06T11:23:52.667 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD": { 2021-09-06T11:23:52.667 INFO:teuthology.orchestra.run.smithi094.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-06T11:23:52.667 INFO:teuthology.orchestra.run.smithi094.stdout: "compression_type": "random" 2021-09-06T11:23:52.667 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:23:52.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:23:52.668 INFO:teuthology.orchestra.run.smithi094.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-06T11:23:52.668 INFO:teuthology.orchestra.run.smithi094.stdout: "index_type": 0 2021-09-06T11:23:52.668 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:23:52.668 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:23:52.669 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:23:52.669 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "", 2021-09-06T11:23:52.669 INFO:teuthology.orchestra.run.smithi094.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-06T11:23:52.669 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:23:52.670 DEBUG:tasks.util.rgw: json result: {'id': '99ed0e5e-8448-467f-bbfd-3849514b595a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-06T11:23:52.671 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-09-06T11:23:52.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-09-06T11:23:52.672 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-09-06T11:23:52.708 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:23:52.708 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:23:52.713 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.713+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.714+0000 7fdbe872a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.714+0000 7fdbe872a380 20 realm 2021-09-06T11:23:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.714+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.715+0000 7fdbe872a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.715+0000 7fdbe872a380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-06T11:23:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.715+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.715+0000 7fdbe872a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.715+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.715+0000 7fdbe872a380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:52.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.715+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.716+0000 7fdbe872a380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T11:23:52.718 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.716+0000 7fdbe872a380 20 searching for the correct realm 2021-09-06T11:23:52.727 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.726+0000 7fdbe872a380 20 RGWRados::pool_iterate: got zone_info.99ed0e5e-8448-467f-bbfd-3849514b595a 2021-09-06T11:23:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.726+0000 7fdbe872a380 20 RGWRados::pool_iterate: got zonegroup_info.eb837787-11a2-48e7-9fd9-f26232451aaa 2021-09-06T11:23:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.726+0000 7fdbe872a380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T11:23:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.726+0000 7fdbe872a380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T11:23:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.726+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.726+0000 7fdbe872a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.726+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.727+0000 7fdbe872a380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.727+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.727+0000 7fdbe872a380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:23:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.727+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.727+0000 7fdbe872a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.727+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.727+0000 7fdbe872a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.727+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.728+0000 7fdbe872a380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.737+0000 7fdbe872a380 20 RGWRados::pool_iterate: got zone_info.99ed0e5e-8448-467f-bbfd-3849514b595a 2021-09-06T11:23:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.737+0000 7fdbe872a380 20 RGWRados::pool_iterate: got zonegroup_info.eb837787-11a2-48e7-9fd9-f26232451aaa 2021-09-06T11:23:52.738 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.737+0000 7fdbe872a380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T11:23:52.738 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.737+0000 7fdbe872a380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T11:23:52.738 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.737+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.739 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.737+0000 7fdbe872a380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:52.739 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.737+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.739 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.737+0000 7fdbe872a380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T11:23:52.739 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.737+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.741 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.738+0000 7fdbe872a380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:52.741 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.738+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.741 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.738+0000 7fdbe872a380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:23:52.742 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.738+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.742 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.742 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 zone default found 2021-09-06T11:23:52.742 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 1 Realm: () 2021-09-06T11:23:52.743 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 1 ZoneGroup: default (eb837787-11a2-48e7-9fd9-f26232451aaa) 2021-09-06T11:23:52.743 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 1 Zone: default (99ed0e5e-8448-467f-bbfd-3849514b595a) 2021-09-06T11:23:52.743 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-06T11:23:52.743 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 zonegroup default 2021-09-06T11:23:52.743 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.744 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.744 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.744 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.744 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.745 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.745 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 started sync module instance, tier type = 2021-09-06T11:23:52.745 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.739+0000 7fdbe872a380 20 started zone id=99ed0e5e-8448-467f-bbfd-3849514b595a (name=default) with tier type = 2021-09-06T11:23:52.750 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.750+0000 7fdbe872a380 20 add_watcher() i=0 2021-09-06T11:23:52.750 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.750+0000 7fdbe872a380 20 add_watcher() i=1 2021-09-06T11:23:52.750 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.750+0000 7fdbe872a380 20 add_watcher() i=2 2021-09-06T11:23:52.750 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.750+0000 7fdbe872a380 20 add_watcher() i=3 2021-09-06T11:23:52.751 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.750+0000 7fdbe872a380 20 add_watcher() i=4 2021-09-06T11:23:52.751 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.750+0000 7fdbe872a380 20 add_watcher() i=5 2021-09-06T11:23:52.751 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.750+0000 7fdbe872a380 20 add_watcher() i=6 2021-09-06T11:23:52.751 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.751+0000 7fdbe872a380 20 add_watcher() i=7 2021-09-06T11:23:52.752 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.751+0000 7fdbe872a380 2 all 8 watchers are set, enabling cache 2021-09-06T11:23:52.752 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.752+0000 7fdbad7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T11:23:52.753 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.752+0000 7fdbe872a380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T11:23:52.753 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.752+0000 7fdbe872a380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T11:23:52.756 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.756+0000 7fdbe872a380 5 note: GC not initialized 2021-09-06T11:23:52.757 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.757+0000 7fdb1bfff700 20 reqs_thread_entry: start 2021-09-06T11:23:52.757 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.757+0000 7fdb19ffb700 20 reqs_thread_entry: start 2021-09-06T11:23:52.758 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.757+0000 7fdbe872a380 20 init_complete bucket index max shards: 11 2021-09-06T11:23:52.758 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.757+0000 7fdbe872a380 10 rgw notify: Started notification manager with: 1 workers 2021-09-06T11:23:52.758 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.757+0000 7fdb07fff700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 6 11:24:23 2021 2021-09-06T11:23:52.758 INFO:teuthology.orchestra.run.smithi094.stderr: 2021-09-06T11:23:52.759 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.757+0000 7fdb077fe700 20 reqs_thread_entry: start 2021-09-06T11:23:52.768 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-09-06T11:23:52.768 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.768+0000 7fdbe872a380 20 remove_watcher() i=0 2021-09-06T11:23:52.768 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.768+0000 7fdbe872a380 2 removed watcher, disabling cache 2021-09-06T11:23:52.769 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.769+0000 7fdbe872a380 20 remove_watcher() i=1 2021-09-06T11:23:52.770 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.770+0000 7fdbe872a380 20 remove_watcher() i=2 2021-09-06T11:23:52.771 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.771+0000 7fdbe872a380 20 remove_watcher() i=3 2021-09-06T11:23:52.772 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.772+0000 7fdbe872a380 20 remove_watcher() i=4 2021-09-06T11:23:52.773 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.773+0000 7fdbe872a380 20 remove_watcher() i=5 2021-09-06T11:23:52.774 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.775+0000 7fdbe872a380 20 remove_watcher() i=6 2021-09-06T11:23:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.776+0000 7fdbe872a380 20 remove_watcher() i=7 2021-09-06T11:23:52.776 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.776+0000 7fdbad7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T11:23:52.781 DEBUG:tasks.util.rgw: json result: [] 2021-09-06T11:23:52.781 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T11:23:52.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T11:23:52.782 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-06T11:23:52.863 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:23:52.863 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:23:52.868 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.868+0000 7f9c6cbc0380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T11:23:52.868 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.868+0000 7f9c6cbc0380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T11:23:52.871 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.871+0000 7f9be17fa700 20 reqs_thread_entry: start 2021-09-06T11:23:52.872 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.871+0000 7f9c6cbc0380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-06T11:23:52.872 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.871+0000 7f9c6cbc0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.872 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.872+0000 7f9c6cbc0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:52.873 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.872+0000 7f9c6cbc0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.873 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.873+0000 7f9c6cbc0380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T11:23:52.883 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.883+0000 7f9c6cbc0380 20 RGWRados::pool_iterate: got zone_info.99ed0e5e-8448-467f-bbfd-3849514b595a 2021-09-06T11:23:52.883 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.883+0000 7f9c6cbc0380 20 RGWRados::pool_iterate: got zonegroup_info.eb837787-11a2-48e7-9fd9-f26232451aaa 2021-09-06T11:23:52.883 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.883+0000 7f9c6cbc0380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T11:23:52.884 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.883+0000 7f9c6cbc0380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T11:23:52.884 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.883+0000 7f9c6cbc0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.884 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.883+0000 7f9c6cbc0380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T11:23:52.884 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.883+0000 7f9c6cbc0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.885 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.883+0000 7f9c6cbc0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:52.885 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.883+0000 7f9c6cbc0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.885 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.884+0000 7f9c6cbc0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:52.886 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.884+0000 7f9c6cbc0380 20 rados->read ofs=0 len=0 2021-09-06T11:23:52.886 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:23:52.884+0000 7f9c6cbc0380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:23:52.888 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:23:52.888 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "99ed0e5e-8448-467f-bbfd-3849514b595a", 2021-09-06T11:23:52.888 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default", 2021-09-06T11:23:52.889 INFO:teuthology.orchestra.run.smithi094.stdout: "domain_root": "default.rgw.meta:root", 2021-09-06T11:23:52.889 INFO:teuthology.orchestra.run.smithi094.stdout: "control_pool": "default.rgw.control", 2021-09-06T11:23:52.889 INFO:teuthology.orchestra.run.smithi094.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-06T11:23:52.889 INFO:teuthology.orchestra.run.smithi094.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-06T11:23:52.889 INFO:teuthology.orchestra.run.smithi094.stdout: "log_pool": "default.rgw.log", 2021-09-06T11:23:52.890 INFO:teuthology.orchestra.run.smithi094.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-06T11:23:52.890 INFO:teuthology.orchestra.run.smithi094.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-06T11:23:52.890 INFO:teuthology.orchestra.run.smithi094.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-06T11:23:52.890 INFO:teuthology.orchestra.run.smithi094.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-06T11:23:52.890 INFO:teuthology.orchestra.run.smithi094.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-06T11:23:52.891 INFO:teuthology.orchestra.run.smithi094.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-06T11:23:52.891 INFO:teuthology.orchestra.run.smithi094.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-06T11:23:52.891 INFO:teuthology.orchestra.run.smithi094.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-06T11:23:52.891 INFO:teuthology.orchestra.run.smithi094.stdout: "otp_pool": "default.rgw.otp", 2021-09-06T11:23:52.892 INFO:teuthology.orchestra.run.smithi094.stdout: "system_key": { 2021-09-06T11:23:52.892 INFO:teuthology.orchestra.run.smithi094.stdout: "access_key": "", 2021-09-06T11:23:52.892 INFO:teuthology.orchestra.run.smithi094.stdout: "secret_key": "" 2021-09-06T11:23:52.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:23:52.892 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_pools": [ 2021-09-06T11:23:52.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:23:52.893 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "default-placement", 2021-09-06T11:23:52.893 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:23:52.893 INFO:teuthology.orchestra.run.smithi094.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-06T11:23:52.893 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": { 2021-09-06T11:23:52.893 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD": { 2021-09-06T11:23:52.894 INFO:teuthology.orchestra.run.smithi094.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-06T11:23:52.894 INFO:teuthology.orchestra.run.smithi094.stdout: "compression_type": "random" 2021-09-06T11:23:52.894 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:23:52.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:23:52.894 INFO:teuthology.orchestra.run.smithi094.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-06T11:23:52.895 INFO:teuthology.orchestra.run.smithi094.stdout: "index_type": 0 2021-09-06T11:23:52.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:23:52.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:23:52.895 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:23:52.895 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "", 2021-09-06T11:23:52.896 INFO:teuthology.orchestra.run.smithi094.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-06T11:23:52.896 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:23:52.897 DEBUG:tasks.util.rgw: json result: {'id': '99ed0e5e-8448-467f-bbfd-3849514b595a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-06T11:23:52.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-09-06T11:23:52.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-09-06T11:23:52.898 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-09-06T11:23:53.013 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:23:53.013 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:23:53.020 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:53.020+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.130 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.130+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.130 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.130+0000 7f2dbfadf380 20 realm 2021-09-06T11:23:54.130 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.130+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.131 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.130+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.131 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.131+0000 7f2dbfadf380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-06T11:23:54.131 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.131+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.132 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.131+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.132 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.131+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.132 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.131+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.132 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.131+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.133 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.132+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.133 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.132+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.134 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.135+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.135 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.135+0000 7f2dbfadf380 10 Creating default zonegroup 2021-09-06T11:23:54.135 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.135+0000 7f2dbfadf380 20 get_system_obj_state: rctx=0x7ffc04d56580 obj=:.pools.avail state=0x5558bcb2ed70 s->prefetch_data=0 2021-09-06T11:23:54.135 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.135+0000 7f2dbfadf380 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-06T11:23:54.153 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.153+0000 7f2dbfadf380 10 WARNING: store->list_zones() returned r=-2 2021-09-06T11:23:54.153 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.153+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.153 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.153+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.157 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.157+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.158 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.158+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.158 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.158+0000 7f2dbfadf380 10 could not read realm id: (2) No such file or directory 2021-09-06T11:23:54.158 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.158+0000 7f2dbfadf380 10 WARNING: failed to set zone as default, r=-22 2021-09-06T11:23:54.159 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.158+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.159 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.158+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.162 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.163+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.163 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.163+0000 7f2dbfadf380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T11:23:54.163 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.163+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.165 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.164+0000 7f2dbfadf380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:23:54.165 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.164+0000 7f2dbfadf380 10 Using default name default 2021-09-06T11:23:54.165 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.164+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.165 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.164+0000 7f2dbfadf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:54.166 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.164+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.166 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.165+0000 7f2dbfadf380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T11:23:54.166 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.165+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.166 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.165+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.167 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.165+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.168 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.165+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.168 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.165+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.168 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.166+0000 7f2dbfadf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:54.176 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.177+0000 7f2dbfadf380 20 RGWRados::pool_iterate: got zone_info.d56f2efd-a6bd-4a7e-b583-cee742c358cc 2021-09-06T11:23:54.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.177+0000 7f2dbfadf380 20 RGWRados::pool_iterate: got zonegroup_info.63586ec5-fc30-429e-9ae5-6cec744ca12f 2021-09-06T11:23:54.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.177+0000 7f2dbfadf380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T11:23:54.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.177+0000 7f2dbfadf380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T11:23:54.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.177+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.177+0000 7f2dbfadf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:54.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.177+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.177+0000 7f2dbfadf380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T11:23:54.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.177+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:23:54.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:23:54.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 20 zone default found 2021-09-06T11:23:54.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 1 Realm: () 2021-09-06T11:23:54.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 1 ZoneGroup: default (63586ec5-fc30-429e-9ae5-6cec744ca12f) 2021-09-06T11:23:54.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 1 Zone: default (d56f2efd-a6bd-4a7e-b583-cee742c358cc) 2021-09-06T11:23:54.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-06T11:23:54.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 20 zonegroup default 2021-09-06T11:23:54.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.178+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:54.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.179+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:54.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:54.179+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:56.147 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:56.147+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:56.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:56.147+0000 7f2dbfadf380 20 rados->read ofs=0 len=0 2021-09-06T11:23:56.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:56.149+0000 7f2dbfadf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:23:56.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:56.149+0000 7f2dbfadf380 20 started sync module instance, tier type = 2021-09-06T11:23:56.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:56.149+0000 7f2dbfadf380 20 started zone id=d56f2efd-a6bd-4a7e-b583-cee742c358cc (name=default) with tier type = 2021-09-06T11:23:58.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.185+0000 7f2dbfadf380 20 add_watcher() i=0 2021-09-06T11:23:58.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.185+0000 7f2dbfadf380 20 add_watcher() i=1 2021-09-06T11:23:58.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.185+0000 7f2dbfadf380 20 add_watcher() i=2 2021-09-06T11:23:58.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.185+0000 7f2dbfadf380 20 add_watcher() i=3 2021-09-06T11:23:58.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.185+0000 7f2dbfadf380 20 add_watcher() i=4 2021-09-06T11:23:58.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.185+0000 7f2dbfadf380 20 add_watcher() i=5 2021-09-06T11:23:58.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.185+0000 7f2dbfadf380 20 add_watcher() i=6 2021-09-06T11:23:58.188 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.186+0000 7f2dbfadf380 20 add_watcher() i=7 2021-09-06T11:23:58.188 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.186+0000 7f2dbfadf380 2 all 8 watchers are set, enabling cache 2021-09-06T11:23:58.188 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.186+0000 7f2dbfadf380 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-06T11:23:58.188 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.186+0000 7f2dbfadf380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-06T11:23:58.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.186+0000 7f2dbfadf380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T11:23:58.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.187+0000 7f2dbfadf380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-06T11:23:58.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.187+0000 7f2dbfadf380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-06T11:23:58.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.187+0000 7f2dbfadf380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T11:23:58.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.187+0000 7f2dbfadf380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-06T11:23:58.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.187+0000 7f2dbfadf380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-06T11:23:58.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.187+0000 7f2dbfadf380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T11:23:58.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.187+0000 7f2dbfadf380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-06T11:23:58.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.187+0000 7f2dbfadf380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-06T11:23:58.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.187+0000 7f2dbfadf380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T11:23:58.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.188+0000 7f2dbfadf380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-06T11:23:58.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.188+0000 7f2dbfadf380 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-06T11:23:58.192 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.189+0000 7f2dbfadf380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T11:23:58.192 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.192+0000 7f2cfffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T11:23:58.192 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.192+0000 7f2dbfadf380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T11:23:58.193 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:23:58.192+0000 7f2dbfadf380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T11:24:02.167 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.165+0000 7f2dbfadf380 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-06T11:24:02.167 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.165+0000 7f2dbfadf380 5 note: GC not initialized 2021-09-06T11:24:02.167 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.167+0000 7f2cf2ffd700 20 reqs_thread_entry: start 2021-09-06T11:24:02.168 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.167+0000 7f2cf0ff9700 20 reqs_thread_entry: start 2021-09-06T11:24:02.168 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.167+0000 7f2dbfadf380 20 init_complete bucket index max shards: 11 2021-09-06T11:24:02.169 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.167+0000 7f2dbfadf380 10 rgw notify: Started notification manager with: 1 workers 2021-09-06T11:24:02.169 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.167+0000 7f2cd6ffd700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 6 11:24:32 2021 2021-09-06T11:24:02.169 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-09-06T11:24:02.169 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.168+0000 7f2cd67fc700 20 reqs_thread_entry: start 2021-09-06T11:24:02.189 INFO:teuthology.orchestra.run.smithi139.stdout:[] 2021-09-06T11:24:02.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.190+0000 7f2dbfadf380 20 remove_watcher() i=0 2021-09-06T11:24:02.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.190+0000 7f2dbfadf380 2 removed watcher, disabling cache 2021-09-06T11:24:02.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.191+0000 7f2dbfadf380 20 remove_watcher() i=1 2021-09-06T11:24:02.193 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.194+0000 7f2dbfadf380 20 remove_watcher() i=2 2021-09-06T11:24:02.196 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.196+0000 7f2dbfadf380 20 remove_watcher() i=3 2021-09-06T11:24:02.197 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.198+0000 7f2dbfadf380 20 remove_watcher() i=4 2021-09-06T11:24:02.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.199+0000 7f2dbfadf380 20 remove_watcher() i=5 2021-09-06T11:24:02.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.202+0000 7f2dbfadf380 20 remove_watcher() i=6 2021-09-06T11:24:02.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.203+0000 7f2dbfadf380 20 remove_watcher() i=7 2021-09-06T11:24:02.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.204+0000 7f2cfffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T11:24:02.209 DEBUG:tasks.util.rgw: json result: [] 2021-09-06T11:24:02.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T11:24:02.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T11:24:02.210 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-06T11:24:02.255 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:02.256 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:02.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.262+0000 7f896c9ae380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T11:24:02.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.262+0000 7f896c9ae380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T11:24:02.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.265+0000 7f88f17fa700 20 reqs_thread_entry: start 2021-09-06T11:24:02.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.265+0000 7f896c9ae380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-06T11:24:02.266 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.265+0000 7f896c9ae380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.267 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.266+0000 7f896c9ae380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:24:02.267 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.266+0000 7f896c9ae380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.267 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.267+0000 7f896c9ae380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T11:24:02.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.277+0000 7f896c9ae380 20 RGWRados::pool_iterate: got zone_info.d56f2efd-a6bd-4a7e-b583-cee742c358cc 2021-09-06T11:24:02.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.277+0000 7f896c9ae380 20 RGWRados::pool_iterate: got zonegroup_info.63586ec5-fc30-429e-9ae5-6cec744ca12f 2021-09-06T11:24:02.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.277+0000 7f896c9ae380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T11:24:02.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.277+0000 7f896c9ae380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T11:24:02.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.277+0000 7f896c9ae380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.278+0000 7f896c9ae380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T11:24:02.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.278+0000 7f896c9ae380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.278+0000 7f896c9ae380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.278+0000 7f896c9ae380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.278+0000 7f896c9ae380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:24:02.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.278+0000 7f896c9ae380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.279+0000 7f896c9ae380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:24:02.283 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:24:02.283 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "d56f2efd-a6bd-4a7e-b583-cee742c358cc", 2021-09-06T11:24:02.283 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default", 2021-09-06T11:24:02.283 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "default.rgw.meta:root", 2021-09-06T11:24:02.284 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "default.rgw.control", 2021-09-06T11:24:02.284 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-06T11:24:02.284 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-06T11:24:02.284 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "default.rgw.log", 2021-09-06T11:24:02.285 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-06T11:24:02.285 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-06T11:24:02.285 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-06T11:24:02.285 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-06T11:24:02.285 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-06T11:24:02.286 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-06T11:24:02.286 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-06T11:24:02.286 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-06T11:24:02.286 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "default.rgw.otp", 2021-09-06T11:24:02.287 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2021-09-06T11:24:02.287 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "", 2021-09-06T11:24:02.287 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "" 2021-09-06T11:24:02.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:02.288 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2021-09-06T11:24:02.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:02.288 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2021-09-06T11:24:02.288 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:24:02.289 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-06T11:24:02.289 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2021-09-06T11:24:02.289 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2021-09-06T11:24:02.289 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-06T11:24:02.289 INFO:teuthology.orchestra.run.smithi139.stdout: "compression_type": "random" 2021-09-06T11:24:02.290 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:02.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:02.290 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-06T11:24:02.290 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0 2021-09-06T11:24:02.291 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:02.291 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:02.291 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:02.291 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "", 2021-09-06T11:24:02.292 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-06T11:24:02.292 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:24:02.293 DEBUG:tasks.util.rgw: json result: {'id': 'd56f2efd-a6bd-4a7e-b583-cee742c358cc', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-06T11:24:02.293 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-09-06T11:24:02.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-09-06T11:24:02.294 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-09-06T11:24:02.331 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:02.332 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:02.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.337+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.338+0000 7f97e479b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.338+0000 7f97e479b380 20 realm 2021-09-06T11:24:02.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.338+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.338+0000 7f97e479b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.338+0000 7f97e479b380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-06T11:24:02.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.338+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.339+0000 7f97e479b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.339+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.339+0000 7f97e479b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:24:02.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.339+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.340+0000 7f97e479b380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T11:24:02.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.340+0000 7f97e479b380 20 searching for the correct realm 2021-09-06T11:24:02.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.350+0000 7f97e479b380 20 RGWRados::pool_iterate: got zone_info.d56f2efd-a6bd-4a7e-b583-cee742c358cc 2021-09-06T11:24:02.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.350+0000 7f97e479b380 20 RGWRados::pool_iterate: got zonegroup_info.63586ec5-fc30-429e-9ae5-6cec744ca12f 2021-09-06T11:24:02.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.350+0000 7f97e479b380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T11:24:02.351 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.350+0000 7f97e479b380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T11:24:02.351 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.350+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.351 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.350+0000 7f97e479b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.351 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.350+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.352 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.350+0000 7f97e479b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:24:02.352 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.350+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.353 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.351+0000 7f97e479b380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:24:02.353 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.351+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.353 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.351+0000 7f97e479b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.354 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.351+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.354 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.351+0000 7f97e479b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.354 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.351+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.354 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.352+0000 7f97e479b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:24:02.361 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.360+0000 7f97e479b380 20 RGWRados::pool_iterate: got zone_info.d56f2efd-a6bd-4a7e-b583-cee742c358cc 2021-09-06T11:24:02.361 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.360+0000 7f97e479b380 20 RGWRados::pool_iterate: got zonegroup_info.63586ec5-fc30-429e-9ae5-6cec744ca12f 2021-09-06T11:24:02.361 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.360+0000 7f97e479b380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T11:24:02.361 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.360+0000 7f97e479b380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T11:24:02.362 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.360+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.362 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.361+0000 7f97e479b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:24:02.362 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.361+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.362 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.361+0000 7f97e479b380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T11:24:02.363 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.361+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.364 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.361+0000 7f97e479b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:24:02.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.361+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:24:02.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 20 zone default found 2021-09-06T11:24:02.366 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 1 Realm: () 2021-09-06T11:24:02.366 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 1 ZoneGroup: default (63586ec5-fc30-429e-9ae5-6cec744ca12f) 2021-09-06T11:24:02.366 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 1 Zone: default (d56f2efd-a6bd-4a7e-b583-cee742c358cc) 2021-09-06T11:24:02.366 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-06T11:24:02.367 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 20 zonegroup default 2021-09-06T11:24:02.367 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.367 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.367 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.362+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.367 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.363+0000 7f97e479b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.368 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.363+0000 7f97e479b380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.368 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.363+0000 7f97e479b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.368 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.363+0000 7f97e479b380 20 started sync module instance, tier type = 2021-09-06T11:24:02.368 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.363+0000 7f97e479b380 20 started zone id=d56f2efd-a6bd-4a7e-b583-cee742c358cc (name=default) with tier type = 2021-09-06T11:24:02.390 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.391+0000 7f97e479b380 20 add_watcher() i=0 2021-09-06T11:24:02.391 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.391+0000 7f97e479b380 20 add_watcher() i=1 2021-09-06T11:24:02.391 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.391+0000 7f97e479b380 20 add_watcher() i=2 2021-09-06T11:24:02.391 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.391+0000 7f97e479b380 20 add_watcher() i=3 2021-09-06T11:24:02.391 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.391+0000 7f97e479b380 20 add_watcher() i=4 2021-09-06T11:24:02.391 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.391+0000 7f97e479b380 20 add_watcher() i=5 2021-09-06T11:24:02.392 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.391+0000 7f97e479b380 20 add_watcher() i=6 2021-09-06T11:24:02.393 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.393+0000 7f97e479b380 20 add_watcher() i=7 2021-09-06T11:24:02.393 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.393+0000 7f97e479b380 2 all 8 watchers are set, enabling cache 2021-09-06T11:24:02.395 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.395+0000 7f97a97e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T11:24:02.396 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.396+0000 7f97e479b380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T11:24:02.396 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.396+0000 7f97e479b380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T11:24:02.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.399+0000 7f97e479b380 5 note: GC not initialized 2021-09-06T11:24:02.400 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.400+0000 7f9717fff700 20 reqs_thread_entry: start 2021-09-06T11:24:02.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.400+0000 7f9715ffb700 20 reqs_thread_entry: start 2021-09-06T11:24:02.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.400+0000 7f97e479b380 20 init_complete bucket index max shards: 11 2021-09-06T11:24:02.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.400+0000 7f97e479b380 10 rgw notify: Started notification manager with: 1 workers 2021-09-06T11:24:02.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.400+0000 7f9703fff700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 6 11:24:32 2021 2021-09-06T11:24:02.402 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-09-06T11:24:02.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.400+0000 7f97037fe700 20 reqs_thread_entry: start 2021-09-06T11:24:02.411 INFO:teuthology.orchestra.run.smithi139.stdout:[] 2021-09-06T11:24:02.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.411+0000 7f97e479b380 20 remove_watcher() i=0 2021-09-06T11:24:02.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.411+0000 7f97e479b380 2 removed watcher, disabling cache 2021-09-06T11:24:02.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.413+0000 7f97e479b380 20 remove_watcher() i=1 2021-09-06T11:24:02.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.416+0000 7f97e479b380 20 remove_watcher() i=2 2021-09-06T11:24:02.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.419+0000 7f97e479b380 20 remove_watcher() i=3 2021-09-06T11:24:02.421 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.421+0000 7f97e479b380 20 remove_watcher() i=4 2021-09-06T11:24:02.423 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.424+0000 7f97e479b380 20 remove_watcher() i=5 2021-09-06T11:24:02.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.425+0000 7f97e479b380 20 remove_watcher() i=6 2021-09-06T11:24:02.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.427+0000 7f97e479b380 20 remove_watcher() i=7 2021-09-06T11:24:02.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.427+0000 7f97a97e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T11:24:02.431 DEBUG:tasks.util.rgw: json result: [] 2021-09-06T11:24:02.432 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T11:24:02.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T11:24:02.433 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-06T11:24:02.513 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:02.514 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:02.519 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.519+0000 7f5968073380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T11:24:02.519 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.519+0000 7f5968073380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T11:24:02.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.522+0000 7f58a3fff700 20 reqs_thread_entry: start 2021-09-06T11:24:02.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.522+0000 7f5968073380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-06T11:24:02.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.522+0000 7f5968073380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.523+0000 7f5968073380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:24:02.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.523+0000 7f5968073380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.524+0000 7f5968073380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T11:24:02.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.535+0000 7f5968073380 20 RGWRados::pool_iterate: got zone_info.d56f2efd-a6bd-4a7e-b583-cee742c358cc 2021-09-06T11:24:02.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.535+0000 7f5968073380 20 RGWRados::pool_iterate: got zonegroup_info.63586ec5-fc30-429e-9ae5-6cec744ca12f 2021-09-06T11:24:02.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.535+0000 7f5968073380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T11:24:02.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.535+0000 7f5968073380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T11:24:02.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.535+0000 7f5968073380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.536+0000 7f5968073380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T11:24:02.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.536+0000 7f5968073380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.536+0000 7f5968073380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T11:24:02.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.536+0000 7f5968073380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.536+0000 7f5968073380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T11:24:02.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.536+0000 7f5968073380 20 rados->read ofs=0 len=0 2021-09-06T11:24:02.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:02.537+0000 7f5968073380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T11:24:02.540 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:24:02.541 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "d56f2efd-a6bd-4a7e-b583-cee742c358cc", 2021-09-06T11:24:02.541 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default", 2021-09-06T11:24:02.541 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "default.rgw.meta:root", 2021-09-06T11:24:02.541 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "default.rgw.control", 2021-09-06T11:24:02.541 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-06T11:24:02.542 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-06T11:24:02.542 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "default.rgw.log", 2021-09-06T11:24:02.542 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-06T11:24:02.542 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-06T11:24:02.543 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-06T11:24:02.543 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-06T11:24:02.543 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-06T11:24:02.543 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-06T11:24:02.544 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-06T11:24:02.544 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-06T11:24:02.544 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "default.rgw.otp", 2021-09-06T11:24:02.544 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2021-09-06T11:24:02.545 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "", 2021-09-06T11:24:02.545 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "" 2021-09-06T11:24:02.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:02.545 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2021-09-06T11:24:02.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:02.546 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2021-09-06T11:24:02.546 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:24:02.546 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-06T11:24:02.547 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2021-09-06T11:24:02.547 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2021-09-06T11:24:02.547 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-06T11:24:02.547 INFO:teuthology.orchestra.run.smithi139.stdout: "compression_type": "random" 2021-09-06T11:24:02.548 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:02.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:02.548 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-06T11:24:02.548 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0 2021-09-06T11:24:02.549 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:02.549 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:02.549 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:02.549 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "", 2021-09-06T11:24:02.550 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-06T11:24:02.550 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:24:02.551 DEBUG:tasks.util.rgw: json result: {'id': 'd56f2efd-a6bd-4a7e-b583-cee742c358cc', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-06T11:24:02.552 INFO:tasks.rgw:Starting rgw... 2021-09-06T11:24:02.552 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-09-06T11:24:02.552 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-06T11:24:02.553 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-06T11:24:02.554 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-06T11:24:02.557 INFO:tasks.rgw.c1.client.0:Started 2021-09-06T11:24:02.557 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-09-06T11:24:02.558 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-06T11:24:02.558 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-06T11:24:02.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-06T11:24:02.561 INFO:tasks.rgw.c1.client.1:Started 2021-09-06T11:24:02.561 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-09-06T11:24:02.561 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-06T11:24:02.562 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-06T11:24:02.562 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-06T11:24:02.594 INFO:tasks.rgw.c2.client.0:Started 2021-09-06T11:24:02.594 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-09-06T11:24:02.595 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-06T11:24:02.595 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-06T11:24:02.596 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-06T11:24:02.598 INFO:tasks.rgw.c2.client.1:Started 2021-09-06T11:24:02.598 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi094.front.sepia.ceph.com:8000/ 2021-09-06T11:24:02.599 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8000/ 2021-09-06T11:24:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-06T11:24:02.651 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:02.652 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:02.652 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi094.front.sepia.ceph.com port 8000: Connection refused 2021-09-06T11:24:03.653 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8000/ 2021-09-06T11:24:03.672 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:03.672 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:03.673 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-06T11:24:03.675 INFO:teuthology.orchestra.run.smithi094.stdout:anonymous 2021-09-06T11:24:03.675 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi094.front.sepia.ceph.com:8001/ 2021-09-06T11:24:03.675 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8001/ 2021-09-06T11:24:03.732 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:03.732 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:03.733 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-06T11:24:03.734 INFO:teuthology.orchestra.run.smithi094.stdout:anonymous 2021-09-06T11:24:03.735 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi139.front.sepia.ceph.com:8000/ 2021-09-06T11:24:03.735 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8000/ 2021-09-06T11:24:03.776 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:03.776 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:03.777 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-06T11:24:03.778 INFO:teuthology.orchestra.run.smithi139.stdout:anonymous 2021-09-06T11:24:03.778 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi139.front.sepia.ceph.com:8001/ 2021-09-06T11:24:03.779 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8001/ 2021-09-06T11:24:03.836 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:03.837 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:03.838 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-06T11:24:03.839 INFO:teuthology.orchestra.run.smithi139.stdout:anonymous 2021-09-06T11:24:03.839 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-09-06T11:24:03.960 INFO:tasks.rgw_multisite:creating realm.. 2021-09-06T11:24:03.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:03.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:03.962 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:04.000 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:04.000 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:04.033 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:24:04.033 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-realm", 2021-09-06T11:24:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "current_period": "8245bc5e-afe7-418b-afcf-7fd599759612", 2021-09-06T11:24:04.034 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1 2021-09-06T11:24:04.034 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:24:04.036 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-09-06T11:24:04.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:04.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:04.037 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:04.115 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:04.116 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:04.127 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:24:04.128+0000 7f821875f380 -1 old format 2021-09-06T11:24:04.135 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:24:04.135 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "test-zonegroup", 2021-09-06T11:24:04.135 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zonegroup", 2021-09-06T11:24:04.135 INFO:teuthology.orchestra.run.smithi094.stdout: "api_name": "", 2021-09-06T11:24:04.136 INFO:teuthology.orchestra.run.smithi094.stdout: "is_master": "true", 2021-09-06T11:24:04.136 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T11:24:04.136 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:24:04.136 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:04.137 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames": [], 2021-09-06T11:24:04.137 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames_s3website": [], 2021-09-06T11:24:04.137 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "", 2021-09-06T11:24:04.137 INFO:teuthology.orchestra.run.smithi094.stdout: "zones": [], 2021-09-06T11:24:04.137 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_targets": [], 2021-09-06T11:24:04.138 INFO:teuthology.orchestra.run.smithi094.stdout: "default_placement": "", 2021-09-06T11:24:04.138 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:04.138 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_policy": { 2021-09-06T11:24:04.138 INFO:teuthology.orchestra.run.smithi094.stdout: "groups": [] 2021-09-06T11:24:04.139 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:04.139 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:24:04.140 INFO:tasks.rgw_multisite:creating master zone.. 2021-09-06T11:24:04.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--endpoints', 'http://smithi094.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:04.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--endpoints', 'http://smithi094.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:04.141 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key 1XCNWZEYEDFPNNWL --secret s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C --endpoints http://smithi094.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:04.228 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:04.229 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:04.262 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:24:04.262 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:04.262 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone1", 2021-09-06T11:24:04.262 INFO:teuthology.orchestra.run.smithi094.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-06T11:24:04.263 INFO:teuthology.orchestra.run.smithi094.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-06T11:24:04.263 INFO:teuthology.orchestra.run.smithi094.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-06T11:24:04.263 INFO:teuthology.orchestra.run.smithi094.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-06T11:24:04.263 INFO:teuthology.orchestra.run.smithi094.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-06T11:24:04.264 INFO:teuthology.orchestra.run.smithi094.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-06T11:24:04.264 INFO:teuthology.orchestra.run.smithi094.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-06T11:24:04.264 INFO:teuthology.orchestra.run.smithi094.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-06T11:24:04.264 INFO:teuthology.orchestra.run.smithi094.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-06T11:24:04.264 INFO:teuthology.orchestra.run.smithi094.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-06T11:24:04.265 INFO:teuthology.orchestra.run.smithi094.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-06T11:24:04.265 INFO:teuthology.orchestra.run.smithi094.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-06T11:24:04.265 INFO:teuthology.orchestra.run.smithi094.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-06T11:24:04.265 INFO:teuthology.orchestra.run.smithi094.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-06T11:24:04.266 INFO:teuthology.orchestra.run.smithi094.stdout: "system_key": { 2021-09-06T11:24:04.266 INFO:teuthology.orchestra.run.smithi094.stdout: "access_key": "1XCNWZEYEDFPNNWL", 2021-09-06T11:24:04.266 INFO:teuthology.orchestra.run.smithi094.stdout: "secret_key": "s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C" 2021-09-06T11:24:04.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:04.266 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_pools": [ 2021-09-06T11:24:04.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:04.267 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "default-placement", 2021-09-06T11:24:04.267 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:24:04.267 INFO:teuthology.orchestra.run.smithi094.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-06T11:24:04.268 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": { 2021-09-06T11:24:04.268 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD": { 2021-09-06T11:24:04.268 INFO:teuthology.orchestra.run.smithi094.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2021-09-06T11:24:04.268 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:04.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:04.269 INFO:teuthology.orchestra.run.smithi094.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-06T11:24:04.269 INFO:teuthology.orchestra.run.smithi094.stdout: "index_type": 0 2021-09-06T11:24:04.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:04.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:04.270 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:04.270 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:04.270 INFO:teuthology.orchestra.run.smithi094.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-06T11:24:04.270 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:24:04.271 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-09-06T11:24:05.573 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-09-06T11:24:05.586 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-09-06T11:24:06.574 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-09-06T11:24:06.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:06.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:06.654 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:06.700 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:06.700 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:06.727 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:24:06.727 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone1", 2021-09-06T11:24:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-06T11:24:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-06T11:24:06.728 INFO:teuthology.orchestra.run.smithi094.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-06T11:24:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-06T11:24:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-06T11:24:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-06T11:24:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-06T11:24:06.729 INFO:teuthology.orchestra.run.smithi094.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-06T11:24:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-06T11:24:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-06T11:24:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-06T11:24:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-06T11:24:06.730 INFO:teuthology.orchestra.run.smithi094.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-06T11:24:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-06T11:24:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "system_key": { 2021-09-06T11:24:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "access_key": "1XCNWZEYEDFPNNWL", 2021-09-06T11:24:06.731 INFO:teuthology.orchestra.run.smithi094.stdout: "secret_key": "s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C" 2021-09-06T11:24:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_pools": [ 2021-09-06T11:24:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:06.732 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "default-placement", 2021-09-06T11:24:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:24:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-06T11:24:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": { 2021-09-06T11:24:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD": { 2021-09-06T11:24:06.733 INFO:teuthology.orchestra.run.smithi094.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2021-09-06T11:24:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: "compression_type": "random" 2021-09-06T11:24:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:06.734 INFO:teuthology.orchestra.run.smithi094.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-06T11:24:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: "index_type": 0 2021-09-06T11:24:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:06.735 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "", 2021-09-06T11:24:06.736 INFO:teuthology.orchestra.run.smithi094.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-06T11:24:06.736 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:24:06.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:06.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:06.737 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:06.778 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:06.778 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:06.787 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:24:06.787+0000 7f3171300380 0 period (8245bc5e-afe7-418b-afcf-7fd599759612 does not have zone 7556a536-76a8-4c42-adbe-379e64c0e1a8 configured 2021-09-06T11:24:14.667 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:24:14.668 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:24:14.668 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 1, 2021-09-06T11:24:14.668 INFO:teuthology.orchestra.run.smithi094.stdout: "predecessor_uuid": "8245bc5e-afe7-418b-afcf-7fd599759612", 2021-09-06T11:24:14.669 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": [], 2021-09-06T11:24:14.669 INFO:teuthology.orchestra.run.smithi094.stdout: "period_map": { 2021-09-06T11:24:14.669 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:24:14.669 INFO:teuthology.orchestra.run.smithi094.stdout: "zonegroups": [ 2021-09-06T11:24:14.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:14.670 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "test-zonegroup", 2021-09-06T11:24:14.670 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zonegroup", 2021-09-06T11:24:14.670 INFO:teuthology.orchestra.run.smithi094.stdout: "api_name": "", 2021-09-06T11:24:14.670 INFO:teuthology.orchestra.run.smithi094.stdout: "is_master": "true", 2021-09-06T11:24:14.671 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T11:24:14.671 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:24:14.671 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:14.671 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames": [], 2021-09-06T11:24:14.671 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames_s3website": [], 2021-09-06T11:24:14.672 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:14.672 INFO:teuthology.orchestra.run.smithi094.stdout: "zones": [ 2021-09-06T11:24:14.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:14.672 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:14.673 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone1", 2021-09-06T11:24:14.673 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T11:24:14.673 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:24:14.673 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:14.674 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T11:24:14.674 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "false", 2021-09-06T11:24:14.674 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:24:14.675 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T11:24:14.675 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T11:24:14.675 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "true", 2021-09-06T11:24:14.676 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T11:24:14.676 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T11:24:14.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:14.676 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:14.676 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_targets": [ 2021-09-06T11:24:14.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:14.677 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default-placement", 2021-09-06T11:24:14.677 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2021-09-06T11:24:14.677 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": [ 2021-09-06T11:24:14.678 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD" 2021-09-06T11:24:14.678 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:24:14.678 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:14.678 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:14.679 INFO:teuthology.orchestra.run.smithi094.stdout: "default_placement": "default-placement", 2021-09-06T11:24:14.679 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:14.679 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_policy": { 2021-09-06T11:24:14.679 INFO:teuthology.orchestra.run.smithi094.stdout: "groups": [] 2021-09-06T11:24:14.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:14.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:14.680 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:14.681 INFO:teuthology.orchestra.run.smithi094.stdout: "short_zone_ids": [ 2021-09-06T11:24:14.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:14.681 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:14.682 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 772393743 2021-09-06T11:24:14.682 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:14.682 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:24:14.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:14.683 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T11:24:14.684 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:14.684 INFO:teuthology.orchestra.run.smithi094.stdout: "period_config": { 2021-09-06T11:24:14.685 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_quota": { 2021-09-06T11:24:14.685 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T11:24:14.685 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T11:24:14.685 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T11:24:14.686 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T11:24:14.686 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T11:24:14.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:14.687 INFO:teuthology.orchestra.run.smithi094.stdout: "user_quota": { 2021-09-06T11:24:14.687 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T11:24:14.687 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T11:24:14.688 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T11:24:14.688 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T11:24:14.689 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T11:24:14.689 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:14.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:14.689 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:14.690 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_name": "test-realm", 2021-09-06T11:24:14.690 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:24:14.690 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:24:14.691 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-09-06T11:24:14.692 INFO:teuthology.orchestra.run:waiting for 300 2021-09-06T11:24:14.732 INFO:tasks.rgw.c1.client.0.smithi094.stdout:2021-09-06T11:24:14.732+0000 7ff51e9d9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 64192) UID: 0 2021-09-06T11:24:14.733 INFO:tasks.rgw.c1.client.0.smithi094.stdout:2021-09-06T11:24:14.732+0000 7ff52f877440 -1 shutting down 2021-09-06T11:24:17.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-06T11:24:20.794 INFO:tasks.rgw.c1.client.0:Stopped 2021-09-06T11:24:20.795 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-06T11:24:20.795 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-06T11:24:20.798 INFO:tasks.rgw.c1.client.0:Started 2021-09-06T11:24:20.799 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8000 2021-09-06T11:24:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-06T11:24:20.820 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:20.821 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:20.821 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi094.front.sepia.ceph.com port 8000: Connection refused 2021-09-06T11:24:21.822 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8000 2021-09-06T11:24:21.842 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:21.843 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:21.843 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-06T11:24:21.844 INFO:teuthology.orchestra.run.smithi094.stdout:anonymous 2021-09-06T11:24:21.844 INFO:tasks.rgw_multisite:creating admin user.. 2021-09-06T11:24:21.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:21.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:21.845 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 1XCNWZEYEDFPNNWL --secret s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:21.933 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:21.980 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:24:21.981 INFO:teuthology.orchestra.run.smithi094.stdout: "user_id": "realm-admin", 2021-09-06T11:24:21.981 INFO:teuthology.orchestra.run.smithi094.stdout: "display_name": "Realm Admin", 2021-09-06T11:24:21.981 INFO:teuthology.orchestra.run.smithi094.stdout: "email": "", 2021-09-06T11:24:21.981 INFO:teuthology.orchestra.run.smithi094.stdout: "suspended": 0, 2021-09-06T11:24:21.982 INFO:teuthology.orchestra.run.smithi094.stdout: "max_buckets": 1000, 2021-09-06T11:24:21.982 INFO:teuthology.orchestra.run.smithi094.stdout: "subusers": [], 2021-09-06T11:24:21.982 INFO:teuthology.orchestra.run.smithi094.stdout: "keys": [ 2021-09-06T11:24:21.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:21.983 INFO:teuthology.orchestra.run.smithi094.stdout: "user": "realm-admin", 2021-09-06T11:24:21.983 INFO:teuthology.orchestra.run.smithi094.stdout: "access_key": "1XCNWZEYEDFPNNWL", 2021-09-06T11:24:21.983 INFO:teuthology.orchestra.run.smithi094.stdout: "secret_key": "s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C" 2021-09-06T11:24:21.983 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:21.984 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:21.984 INFO:teuthology.orchestra.run.smithi094.stdout: "swift_keys": [], 2021-09-06T11:24:21.984 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": [], 2021-09-06T11:24:21.984 INFO:teuthology.orchestra.run.smithi094.stdout: "op_mask": "read, write, delete", 2021-09-06T11:24:21.985 INFO:teuthology.orchestra.run.smithi094.stdout: "system": "true", 2021-09-06T11:24:21.985 INFO:teuthology.orchestra.run.smithi094.stdout: "default_placement": "", 2021-09-06T11:24:21.985 INFO:teuthology.orchestra.run.smithi094.stdout: "default_storage_class": "", 2021-09-06T11:24:21.985 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_tags": [], 2021-09-06T11:24:21.985 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_quota": { 2021-09-06T11:24:21.986 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T11:24:21.986 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T11:24:21.986 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T11:24:21.986 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T11:24:21.987 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T11:24:21.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:21.987 INFO:teuthology.orchestra.run.smithi094.stdout: "user_quota": { 2021-09-06T11:24:21.987 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T11:24:21.987 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T11:24:21.988 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T11:24:21.988 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T11:24:21.988 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T11:24:21.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:21.989 INFO:teuthology.orchestra.run.smithi094.stdout: "temp_url_keys": [], 2021-09-06T11:24:21.989 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "rgw", 2021-09-06T11:24:21.989 INFO:teuthology.orchestra.run.smithi094.stdout: "mfa_ids": [] 2021-09-06T11:24:21.989 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:24:21.989 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:24:21.994 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-06T11:24:21.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:21.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:21.996 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi094.front.sepia.ceph.com:8000 --access-key 1XCNWZEYEDFPNNWL --secret s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:22.048 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:22.048 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:22.137 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:24:22.137 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:22.138 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-realm", 2021-09-06T11:24:22.138 INFO:teuthology.orchestra.run.smithi139.stdout: "current_period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:24:22.138 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 2 2021-09-06T11:24:22.138 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:24:22.139 INFO:tasks.rgw_multisite:creating zone.. 2021-09-06T11:24:22.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--endpoints', 'http://smithi139.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:22.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--endpoints', 'http://smithi139.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:22.141 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key 1XCNWZEYEDFPNNWL --secret s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C --endpoints http://smithi139.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:22.195 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:22.196 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:22.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:22.234+0000 7f7cd4059380 0 failed reading obj info from .rgw.root:zone_info.7556a536-76a8-4c42-adbe-379e64c0e1a8: (2) No such file or directory 2021-09-06T11:24:22.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:22.234+0000 7f7cd4059380 0 WARNING: could not read zone params for zone id=7556a536-76a8-4c42-adbe-379e64c0e1a8 name=test-zone1 2021-09-06T11:24:22.239 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:24:22.239 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:24:22.240 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone2", 2021-09-06T11:24:22.240 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-06T11:24:22.240 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-06T11:24:22.240 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-06T11:24:22.240 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-06T11:24:22.241 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-06T11:24:22.241 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-06T11:24:22.241 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-06T11:24:22.241 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-06T11:24:22.241 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-06T11:24:22.242 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-06T11:24:22.242 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-06T11:24:22.242 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-06T11:24:22.242 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-06T11:24:22.243 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-06T11:24:22.243 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2021-09-06T11:24:22.243 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "1XCNWZEYEDFPNNWL", 2021-09-06T11:24:22.243 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C" 2021-09-06T11:24:22.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:22.244 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2021-09-06T11:24:22.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:22.244 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2021-09-06T11:24:22.244 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:24:22.245 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-06T11:24:22.245 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2021-09-06T11:24:22.245 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2021-09-06T11:24:22.245 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2021-09-06T11:24:22.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:22.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:22.246 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-06T11:24:22.246 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0 2021-09-06T11:24:22.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:22.247 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:22.247 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:22.247 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:22.247 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-06T11:24:22.248 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:24:22.249 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-09-06T11:24:23.203 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-09-06T11:24:23.221 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-09-06T11:24:23.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-09-06T11:24:24.217 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-09-06T11:24:24.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:24.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:24.243 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:24.290 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:24.290 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:24.318 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:24:24.318 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:24:24.318 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone2", 2021-09-06T11:24:24.318 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-06T11:24:24.319 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-06T11:24:24.319 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-06T11:24:24.319 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-06T11:24:24.319 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-06T11:24:24.320 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-06T11:24:24.320 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-06T11:24:24.320 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-06T11:24:24.321 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-06T11:24:24.321 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-06T11:24:24.321 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-06T11:24:24.321 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-06T11:24:24.322 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-06T11:24:24.322 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-06T11:24:24.322 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2021-09-06T11:24:24.322 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "1XCNWZEYEDFPNNWL", 2021-09-06T11:24:24.323 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C" 2021-09-06T11:24:24.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:24.323 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2021-09-06T11:24:24.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:24.324 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2021-09-06T11:24:24.324 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:24:24.324 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-06T11:24:24.324 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2021-09-06T11:24:24.324 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2021-09-06T11:24:24.325 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-06T11:24:24.325 INFO:teuthology.orchestra.run.smithi139.stdout: "compression_type": "random" 2021-09-06T11:24:24.325 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:24.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:24.326 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-06T11:24:24.326 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0 2021-09-06T11:24:24.326 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:24.326 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:24.326 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:24.327 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "", 2021-09-06T11:24:24.327 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-06T11:24:24.327 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:24:24.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:24.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:24.329 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:24.370 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:24.370 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:24.382 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:24.383+0000 7f28f832a380 0 period (06970c73-91a9-4240-80e4-aa07d550699b does not have zone 2f90b352-42d8-4c14-ad62-fe843301400a configured 2021-09-06T11:24:32.273 INFO:teuthology.orchestra.run.smithi139.stderr:Sending period to new master zone 7556a536-76a8-4c42-adbe-379e64c0e1a8 2021-09-06T11:24:32.320 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:24:32.320 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:24:32.321 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 2, 2021-09-06T11:24:32.321 INFO:teuthology.orchestra.run.smithi139.stdout: "predecessor_uuid": "8245bc5e-afe7-418b-afcf-7fd599759612", 2021-09-06T11:24:32.321 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": [], 2021-09-06T11:24:32.321 INFO:teuthology.orchestra.run.smithi139.stdout: "period_map": { 2021-09-06T11:24:32.322 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:24:32.322 INFO:teuthology.orchestra.run.smithi139.stdout: "zonegroups": [ 2021-09-06T11:24:32.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:32.322 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "test-zonegroup", 2021-09-06T11:24:32.322 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zonegroup", 2021-09-06T11:24:32.323 INFO:teuthology.orchestra.run.smithi139.stdout: "api_name": "", 2021-09-06T11:24:32.323 INFO:teuthology.orchestra.run.smithi139.stdout: "is_master": "true", 2021-09-06T11:24:32.323 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:24:32.324 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:24:32.324 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:32.324 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames": [], 2021-09-06T11:24:32.324 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames_s3website": [], 2021-09-06T11:24:32.325 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:32.325 INFO:teuthology.orchestra.run.smithi139.stdout: "zones": [ 2021-09-06T11:24:32.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:32.325 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:24:32.325 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone2", 2021-09-06T11:24:32.326 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:24:32.326 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2021-09-06T11:24:32.326 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:32.326 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2021-09-06T11:24:32.327 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "true", 2021-09-06T11:24:32.327 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:24:32.327 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2021-09-06T11:24:32.327 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2021-09-06T11:24:32.327 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "true", 2021-09-06T11:24:32.328 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2021-09-06T11:24:32.328 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2021-09-06T11:24:32.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:32.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:32.329 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:32.329 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone1", 2021-09-06T11:24:32.329 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:24:32.329 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:24:32.330 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:32.330 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2021-09-06T11:24:32.330 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "true", 2021-09-06T11:24:32.330 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:24:32.330 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2021-09-06T11:24:32.331 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2021-09-06T11:24:32.331 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "true", 2021-09-06T11:24:32.331 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2021-09-06T11:24:32.331 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2021-09-06T11:24:32.332 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:32.332 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:32.332 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_targets": [ 2021-09-06T11:24:32.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:32.332 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default-placement", 2021-09-06T11:24:32.333 INFO:teuthology.orchestra.run.smithi139.stdout: "tags": [], 2021-09-06T11:24:32.333 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": [ 2021-09-06T11:24:32.333 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD" 2021-09-06T11:24:32.333 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:24:32.334 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:32.334 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:32.334 INFO:teuthology.orchestra.run.smithi139.stdout: "default_placement": "default-placement", 2021-09-06T11:24:32.334 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:32.334 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_policy": { 2021-09-06T11:24:32.335 INFO:teuthology.orchestra.run.smithi139.stdout: "groups": [] 2021-09-06T11:24:32.335 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:32.335 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:32.335 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:32.336 INFO:teuthology.orchestra.run.smithi139.stdout: "short_zone_ids": [ 2021-09-06T11:24:32.336 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:32.336 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:24:32.336 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 4134315483 2021-09-06T11:24:32.337 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:32.337 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:32.337 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:32.337 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 772393743 2021-09-06T11:24:32.337 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:32.338 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:24:32.338 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:32.338 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T11:24:32.338 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:32.339 INFO:teuthology.orchestra.run.smithi139.stdout: "period_config": { 2021-09-06T11:24:32.339 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_quota": { 2021-09-06T11:24:32.339 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2021-09-06T11:24:32.339 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2021-09-06T11:24:32.339 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2021-09-06T11:24:32.340 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2021-09-06T11:24:32.340 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2021-09-06T11:24:32.340 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:32.341 INFO:teuthology.orchestra.run.smithi139.stdout: "user_quota": { 2021-09-06T11:24:32.341 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2021-09-06T11:24:32.341 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2021-09-06T11:24:32.341 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2021-09-06T11:24:32.341 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2021-09-06T11:24:32.342 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2021-09-06T11:24:32.342 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:32.342 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:32.342 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:32.343 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_name": "test-realm", 2021-09-06T11:24:32.343 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:24:32.343 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:24:32.344 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-09-06T11:24:32.345 INFO:teuthology.orchestra.run:waiting for 300 2021-09-06T11:24:32.386 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2021-09-06T11:24:32.386+0000 7f568eedb700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 65090) UID: 0 2021-09-06T11:24:32.386 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2021-09-06T11:24:32.386+0000 7f569fd79440 -1 shutting down 2021-09-06T11:24:33.711 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-09-06T11:24:38.446 INFO:tasks.rgw.c2.client.0:Stopped 2021-09-06T11:24:38.447 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-06T11:24:38.447 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-06T11:24:38.450 INFO:tasks.rgw.c2.client.0:Started 2021-09-06T11:24:38.451 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8000 2021-09-06T11:24:38.471 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:38.472 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-06T11:24:38.473 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi139.front.sepia.ceph.com port 8000: Connection refused 2021-09-06T11:24:39.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-09-06T11:24:39.475 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8000 2021-09-06T11:24:39.495 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:39.495 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:39.496 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-06T11:24:39.497 INFO:teuthology.orchestra.run.smithi139.stdout:anonymous 2021-09-06T11:24:39.498 INFO:tasks.rgw_multisite:creating zone.. 2021-09-06T11:24:39.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--endpoints', 'http://smithi094.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:39.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--endpoints', 'http://smithi094.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:39.499 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 1XCNWZEYEDFPNNWL --secret s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C --endpoints http://smithi094.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:39.555 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:39.555 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:39.586 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:24:39.586+0000 7f3694b50380 0 failed reading obj info from .rgw.root:zone_info.2f90b352-42d8-4c14-ad62-fe843301400a: (2) No such file or directory 2021-09-06T11:24:39.587 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:24:39.586+0000 7f3694b50380 0 WARNING: could not read zone params for zone id=2f90b352-42d8-4c14-ad62-fe843301400a name=test-zone2 2021-09-06T11:24:39.591 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:24:39.591 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T11:24:39.592 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone3", 2021-09-06T11:24:39.592 INFO:teuthology.orchestra.run.smithi094.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-06T11:24:39.592 INFO:teuthology.orchestra.run.smithi094.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-06T11:24:39.592 INFO:teuthology.orchestra.run.smithi094.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-06T11:24:39.593 INFO:teuthology.orchestra.run.smithi094.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-06T11:24:39.593 INFO:teuthology.orchestra.run.smithi094.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-06T11:24:39.593 INFO:teuthology.orchestra.run.smithi094.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-06T11:24:39.593 INFO:teuthology.orchestra.run.smithi094.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-06T11:24:39.594 INFO:teuthology.orchestra.run.smithi094.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-06T11:24:39.594 INFO:teuthology.orchestra.run.smithi094.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-06T11:24:39.594 INFO:teuthology.orchestra.run.smithi094.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-06T11:24:39.594 INFO:teuthology.orchestra.run.smithi094.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-06T11:24:39.595 INFO:teuthology.orchestra.run.smithi094.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-06T11:24:39.599 INFO:teuthology.orchestra.run.smithi094.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-06T11:24:39.599 INFO:teuthology.orchestra.run.smithi094.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-06T11:24:39.599 INFO:teuthology.orchestra.run.smithi094.stdout: "system_key": { 2021-09-06T11:24:39.600 INFO:teuthology.orchestra.run.smithi094.stdout: "access_key": "1XCNWZEYEDFPNNWL", 2021-09-06T11:24:39.600 INFO:teuthology.orchestra.run.smithi094.stdout: "secret_key": "s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C" 2021-09-06T11:24:39.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:39.600 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_pools": [ 2021-09-06T11:24:39.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:39.601 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "default-placement", 2021-09-06T11:24:39.601 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:24:39.601 INFO:teuthology.orchestra.run.smithi094.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-06T11:24:39.602 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": { 2021-09-06T11:24:39.602 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD": { 2021-09-06T11:24:39.602 INFO:teuthology.orchestra.run.smithi094.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2021-09-06T11:24:39.602 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:39.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:39.603 INFO:teuthology.orchestra.run.smithi094.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-06T11:24:39.603 INFO:teuthology.orchestra.run.smithi094.stdout: "index_type": 0 2021-09-06T11:24:39.603 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:39.604 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:39.604 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:39.604 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:39.604 INFO:teuthology.orchestra.run.smithi094.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-06T11:24:39.604 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:24:39.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-09-06T11:24:40.630 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-09-06T11:24:40.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-09-06T11:24:41.644 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-09-06T11:24:41.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:41.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:41.663 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:41.712 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:41.713 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:41.739 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:24:41.740 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T11:24:41.740 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone3", 2021-09-06T11:24:41.740 INFO:teuthology.orchestra.run.smithi094.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-06T11:24:41.740 INFO:teuthology.orchestra.run.smithi094.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-06T11:24:41.741 INFO:teuthology.orchestra.run.smithi094.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-06T11:24:41.741 INFO:teuthology.orchestra.run.smithi094.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-06T11:24:41.741 INFO:teuthology.orchestra.run.smithi094.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-06T11:24:41.741 INFO:teuthology.orchestra.run.smithi094.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-06T11:24:41.742 INFO:teuthology.orchestra.run.smithi094.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-06T11:24:41.742 INFO:teuthology.orchestra.run.smithi094.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-06T11:24:41.742 INFO:teuthology.orchestra.run.smithi094.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-06T11:24:41.742 INFO:teuthology.orchestra.run.smithi094.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-06T11:24:41.743 INFO:teuthology.orchestra.run.smithi094.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-06T11:24:41.743 INFO:teuthology.orchestra.run.smithi094.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-06T11:24:41.743 INFO:teuthology.orchestra.run.smithi094.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-06T11:24:41.743 INFO:teuthology.orchestra.run.smithi094.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-06T11:24:41.744 INFO:teuthology.orchestra.run.smithi094.stdout: "system_key": { 2021-09-06T11:24:41.744 INFO:teuthology.orchestra.run.smithi094.stdout: "access_key": "1XCNWZEYEDFPNNWL", 2021-09-06T11:24:41.744 INFO:teuthology.orchestra.run.smithi094.stdout: "secret_key": "s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C" 2021-09-06T11:24:41.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:41.745 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_pools": [ 2021-09-06T11:24:41.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:41.745 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "default-placement", 2021-09-06T11:24:41.746 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:24:41.746 INFO:teuthology.orchestra.run.smithi094.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-06T11:24:41.746 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": { 2021-09-06T11:24:41.746 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD": { 2021-09-06T11:24:41.747 INFO:teuthology.orchestra.run.smithi094.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2021-09-06T11:24:41.747 INFO:teuthology.orchestra.run.smithi094.stdout: "compression_type": "random" 2021-09-06T11:24:41.747 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:41.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:41.747 INFO:teuthology.orchestra.run.smithi094.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-06T11:24:41.748 INFO:teuthology.orchestra.run.smithi094.stdout: "index_type": 0 2021-09-06T11:24:41.748 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:41.748 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:41.748 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:41.749 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "", 2021-09-06T11:24:41.749 INFO:teuthology.orchestra.run.smithi094.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-06T11:24:41.749 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:24:41.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:41.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:41.751 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:41.791 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:41.792 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:41.801 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:24:41.801+0000 7fe72cbae380 0 period (06970c73-91a9-4240-80e4-aa07d550699b does not have zone c248f771-5d9b-48a2-a793-2bc85cc1f251 configured 2021-09-06T11:24:49.694 INFO:teuthology.orchestra.run.smithi094.stderr:Sending period to new master zone 7556a536-76a8-4c42-adbe-379e64c0e1a8 2021-09-06T11:24:49.724 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:24:49.725 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:24:49.725 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 3, 2021-09-06T11:24:49.725 INFO:teuthology.orchestra.run.smithi094.stdout: "predecessor_uuid": "8245bc5e-afe7-418b-afcf-7fd599759612", 2021-09-06T11:24:49.725 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": [], 2021-09-06T11:24:49.726 INFO:teuthology.orchestra.run.smithi094.stdout: "period_map": { 2021-09-06T11:24:49.726 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:24:49.726 INFO:teuthology.orchestra.run.smithi094.stdout: "zonegroups": [ 2021-09-06T11:24:49.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:49.727 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "test-zonegroup", 2021-09-06T11:24:49.727 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zonegroup", 2021-09-06T11:24:49.727 INFO:teuthology.orchestra.run.smithi094.stdout: "api_name": "", 2021-09-06T11:24:49.728 INFO:teuthology.orchestra.run.smithi094.stdout: "is_master": "true", 2021-09-06T11:24:49.728 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T11:24:49.728 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:24:49.728 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:49.729 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames": [], 2021-09-06T11:24:49.729 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames_s3website": [], 2021-09-06T11:24:49.729 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:49.729 INFO:teuthology.orchestra.run.smithi094.stdout: "zones": [ 2021-09-06T11:24:49.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:49.730 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:24:49.730 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone2", 2021-09-06T11:24:49.731 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T11:24:49.731 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2021-09-06T11:24:49.731 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:49.731 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T11:24:49.732 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T11:24:49.732 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:24:49.732 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T11:24:49.732 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T11:24:49.733 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "true", 2021-09-06T11:24:49.733 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T11:24:49.733 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T11:24:49.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:49.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:49.734 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:49.734 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone1", 2021-09-06T11:24:49.734 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T11:24:49.735 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:24:49.735 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:49.735 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T11:24:49.735 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T11:24:49.736 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:24:49.736 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T11:24:49.736 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T11:24:49.736 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "true", 2021-09-06T11:24:49.737 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T11:24:49.737 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T11:24:49.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:49.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:49.738 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T11:24:49.738 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone3", 2021-09-06T11:24:49.739 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T11:24:49.739 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8001" 2021-09-06T11:24:49.739 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:49.739 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T11:24:49.739 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T11:24:49.740 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:24:49.740 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T11:24:49.740 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T11:24:49.740 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "true", 2021-09-06T11:24:49.741 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T11:24:49.741 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T11:24:49.741 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:49.741 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:49.742 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_targets": [ 2021-09-06T11:24:49.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:49.742 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default-placement", 2021-09-06T11:24:49.742 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2021-09-06T11:24:49.742 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": [ 2021-09-06T11:24:49.743 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD" 2021-09-06T11:24:49.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:24:49.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:49.743 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:49.744 INFO:teuthology.orchestra.run.smithi094.stdout: "default_placement": "default-placement", 2021-09-06T11:24:49.744 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:49.744 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_policy": { 2021-09-06T11:24:49.744 INFO:teuthology.orchestra.run.smithi094.stdout: "groups": [] 2021-09-06T11:24:49.744 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:49.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:49.745 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:24:49.745 INFO:teuthology.orchestra.run.smithi094.stdout: "short_zone_ids": [ 2021-09-06T11:24:49.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:49.746 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:24:49.746 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 4134315483 2021-09-06T11:24:49.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:49.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:49.747 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:49.747 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 772393743 2021-09-06T11:24:49.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:49.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:24:49.748 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T11:24:49.748 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 242578107 2021-09-06T11:24:49.748 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:49.748 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:24:49.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:49.749 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T11:24:49.749 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:24:49.749 INFO:teuthology.orchestra.run.smithi094.stdout: "period_config": { 2021-09-06T11:24:49.749 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_quota": { 2021-09-06T11:24:49.750 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T11:24:49.750 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T11:24:49.750 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T11:24:49.750 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T11:24:49.751 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T11:24:49.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:49.751 INFO:teuthology.orchestra.run.smithi094.stdout: "user_quota": { 2021-09-06T11:24:49.751 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T11:24:49.752 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T11:24:49.753 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T11:24:49.753 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T11:24:49.753 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T11:24:49.753 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:24:49.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:24:49.754 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:49.754 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_name": "test-realm", 2021-09-06T11:24:49.754 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:24:49.755 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:24:49.755 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-09-06T11:24:49.756 INFO:teuthology.orchestra.run:waiting for 300 2021-09-06T11:24:49.794 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:24:49.794+0000 7f8b27c34700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 64195) UID: 0 2021-09-06T11:24:49.795 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:24:49.794+0000 7f8b38ad2440 -1 shutting down 2021-09-06T11:24:51.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-09-06T11:24:55.858 INFO:tasks.rgw.c1.client.1:Stopped 2021-09-06T11:24:55.859 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-06T11:24:55.859 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-06T11:24:55.862 INFO:tasks.rgw.c1.client.1:Started 2021-09-06T11:24:55.863 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8001 2021-09-06T11:24:55.884 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:55.884 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:55.884 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi094.front.sepia.ceph.com port 8001: Connection refused 2021-09-06T11:24:55.885 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-06T11:24:56.886 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8001 2021-09-06T11:24:56.906 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:24:56.906 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:24:56.907 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-06T11:24:56.908 INFO:teuthology.orchestra.run.smithi094.stdout:anonymous 2021-09-06T11:24:56.908 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-06T11:24:56.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:56.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:56.910 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi094.front.sepia.ceph.com:8000 --access-key 1XCNWZEYEDFPNNWL --secret s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C --url http://smithi094.front.sepia.ceph.com:8000 --access-key 1XCNWZEYEDFPNNWL --secret s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:56.965 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:56.966 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:57.003 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2021-09-06T11:24:57.002+0000 7f3e2ae4e700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.33774ddf-9586-4cf3-8e79-39a5aa159120.control err=-107 2021-09-06T11:24:57.014 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:24:57.015 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:57.015 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-realm", 2021-09-06T11:24:57.015 INFO:teuthology.orchestra.run.smithi139.stdout: "current_period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:24:57.016 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 2 2021-09-06T11:24:57.016 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:24:57.017 INFO:tasks.rgw_multisite:creating zone.. 2021-09-06T11:24:57.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--endpoints', 'http://smithi139.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:57.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '1XCNWZEYEDFPNNWL', '--secret', 's07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C', '--endpoints', 'http://smithi139.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:57.019 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 1XCNWZEYEDFPNNWL --secret s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C --endpoints http://smithi139.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:57.075 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:57.075 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:57.105 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:57.104+0000 7f81fa9e8380 0 failed reading obj info from .rgw.root:zone_info.7556a536-76a8-4c42-adbe-379e64c0e1a8: (2) No such file or directory 2021-09-06T11:24:57.105 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:57.104+0000 7f81fa9e8380 0 WARNING: could not read zone params for zone id=7556a536-76a8-4c42-adbe-379e64c0e1a8 name=test-zone1 2021-09-06T11:24:57.105 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:57.104+0000 7f81fa9e8380 0 failed reading obj info from .rgw.root:zone_info.c248f771-5d9b-48a2-a793-2bc85cc1f251: (2) No such file or directory 2021-09-06T11:24:57.106 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:57.104+0000 7f81fa9e8380 0 WARNING: could not read zone params for zone id=c248f771-5d9b-48a2-a793-2bc85cc1f251 name=test-zone3 2021-09-06T11:24:57.108 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:24:57.109 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "4487d356-c18b-4713-bd68-02dfa183d02a", 2021-09-06T11:24:57.109 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone4", 2021-09-06T11:24:57.109 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-06T11:24:57.109 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-06T11:24:57.110 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-06T11:24:57.110 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-06T11:24:57.110 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-06T11:24:57.110 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-06T11:24:57.111 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-06T11:24:57.111 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-06T11:24:57.111 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-06T11:24:57.111 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-06T11:24:57.112 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-06T11:24:57.112 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-06T11:24:57.112 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-06T11:24:57.112 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-06T11:24:57.113 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2021-09-06T11:24:57.113 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "1XCNWZEYEDFPNNWL", 2021-09-06T11:24:57.113 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C" 2021-09-06T11:24:57.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:57.114 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2021-09-06T11:24:57.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:57.114 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2021-09-06T11:24:57.114 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:24:57.115 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-06T11:24:57.115 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2021-09-06T11:24:57.115 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2021-09-06T11:24:57.115 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2021-09-06T11:24:57.115 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:57.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:57.116 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-06T11:24:57.116 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0 2021-09-06T11:24:57.116 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:57.117 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:57.117 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:57.117 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_config": { 2021-09-06T11:24:57.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event_retention_days": 7, 2021-09-06T11:24:57.117 INFO:teuthology.orchestra.run.smithi139.stdout: "start_with_full_sync": false 2021-09-06T11:24:57.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:57.118 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:24:57.118 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-06T11:24:57.118 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:24:57.120 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-09-06T11:24:57.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-09-06T11:24:58.266 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-09-06T11:24:58.284 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-09-06T11:24:59.289 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-09-06T11:24:59.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:59.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:59.311 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:59.361 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:59.361 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:59.390 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:24:59.390 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "4487d356-c18b-4713-bd68-02dfa183d02a", 2021-09-06T11:24:59.391 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone4", 2021-09-06T11:24:59.391 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-06T11:24:59.391 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-06T11:24:59.391 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-06T11:24:59.392 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-06T11:24:59.392 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-06T11:24:59.392 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-06T11:24:59.392 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-06T11:24:59.393 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-06T11:24:59.393 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-06T11:24:59.393 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-06T11:24:59.393 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-06T11:24:59.393 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-06T11:24:59.394 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-06T11:24:59.394 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-06T11:24:59.394 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2021-09-06T11:24:59.394 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "1XCNWZEYEDFPNNWL", 2021-09-06T11:24:59.395 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C" 2021-09-06T11:24:59.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:59.395 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2021-09-06T11:24:59.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:24:59.396 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2021-09-06T11:24:59.396 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:24:59.396 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-06T11:24:59.396 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2021-09-06T11:24:59.397 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2021-09-06T11:24:59.397 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2021-09-06T11:24:59.397 INFO:teuthology.orchestra.run.smithi139.stdout: "compression_type": "random" 2021-09-06T11:24:59.397 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:59.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:59.398 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-06T11:24:59.398 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0 2021-09-06T11:24:59.398 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:59.398 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:24:59.399 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:24:59.399 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_config": { 2021-09-06T11:24:59.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event_retention_days": 7, 2021-09-06T11:24:59.399 INFO:teuthology.orchestra.run.smithi139.stdout: "start_with_full_sync": false 2021-09-06T11:24:59.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:24:59.400 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "", 2021-09-06T11:24:59.400 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-06T11:24:59.400 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:24:59.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:59.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:24:59.402 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T11:24:59.443 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:24:59.443 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:24:59.454 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:24:59.453+0000 7ff230c41380 0 period (06970c73-91a9-4240-80e4-aa07d550699b does not have zone 4487d356-c18b-4713-bd68-02dfa183d02a configured 2021-09-06T11:25:07.357 INFO:teuthology.orchestra.run.smithi139.stderr:Sending period to new master zone 7556a536-76a8-4c42-adbe-379e64c0e1a8 2021-09-06T11:25:07.396 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:25:07.396 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:07.397 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 4, 2021-09-06T11:25:07.397 INFO:teuthology.orchestra.run.smithi139.stdout: "predecessor_uuid": "8245bc5e-afe7-418b-afcf-7fd599759612", 2021-09-06T11:25:07.397 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": [], 2021-09-06T11:25:07.397 INFO:teuthology.orchestra.run.smithi139.stdout: "period_map": { 2021-09-06T11:25:07.398 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:07.398 INFO:teuthology.orchestra.run.smithi139.stdout: "zonegroups": [ 2021-09-06T11:25:07.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:07.398 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "test-zonegroup", 2021-09-06T11:25:07.399 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zonegroup", 2021-09-06T11:25:07.399 INFO:teuthology.orchestra.run.smithi139.stdout: "api_name": "", 2021-09-06T11:25:07.399 INFO:teuthology.orchestra.run.smithi139.stdout: "is_master": "true", 2021-09-06T11:25:07.399 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:25:07.399 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:25:07.400 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:25:07.400 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames": [], 2021-09-06T11:25:07.400 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames_s3website": [], 2021-09-06T11:25:07.400 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:25:07.400 INFO:teuthology.orchestra.run.smithi139.stdout: "zones": [ 2021-09-06T11:25:07.401 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:07.401 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:25:07.401 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone2", 2021-09-06T11:25:07.401 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:25:07.401 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2021-09-06T11:25:07.402 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:25:07.402 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2021-09-06T11:25:07.402 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "true", 2021-09-06T11:25:07.402 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:25:07.403 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2021-09-06T11:25:07.403 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2021-09-06T11:25:07.403 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "true", 2021-09-06T11:25:07.403 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2021-09-06T11:25:07.403 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2021-09-06T11:25:07.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:07.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:07.404 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "4487d356-c18b-4713-bd68-02dfa183d02a", 2021-09-06T11:25:07.404 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone4", 2021-09-06T11:25:07.404 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:25:07.405 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2021-09-06T11:25:07.405 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:25:07.405 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2021-09-06T11:25:07.405 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "true", 2021-09-06T11:25:07.406 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:25:07.406 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2021-09-06T11:25:07.406 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "pubsub", 2021-09-06T11:25:07.406 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "false", 2021-09-06T11:25:07.407 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [ 2021-09-06T11:25:07.407 INFO:teuthology.orchestra.run.smithi139.stdout: "test-zone1" 2021-09-06T11:25:07.407 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:25:07.407 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2021-09-06T11:25:07.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:07.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:07.408 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:25:07.408 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone1", 2021-09-06T11:25:07.409 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:25:07.409 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:25:07.409 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:25:07.409 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2021-09-06T11:25:07.409 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "true", 2021-09-06T11:25:07.410 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:25:07.410 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2021-09-06T11:25:07.410 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2021-09-06T11:25:07.410 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "true", 2021-09-06T11:25:07.411 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2021-09-06T11:25:07.411 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2021-09-06T11:25:07.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:07.411 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:07.412 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T11:25:07.412 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone3", 2021-09-06T11:25:07.412 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:25:07.412 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi094.front.sepia.ceph.com:8001" 2021-09-06T11:25:07.412 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:25:07.413 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2021-09-06T11:25:07.413 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "true", 2021-09-06T11:25:07.413 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:25:07.413 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2021-09-06T11:25:07.414 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2021-09-06T11:25:07.414 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "true", 2021-09-06T11:25:07.414 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2021-09-06T11:25:07.414 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2021-09-06T11:25:07.414 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:07.415 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:25:07.415 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_targets": [ 2021-09-06T11:25:07.415 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:07.415 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default-placement", 2021-09-06T11:25:07.416 INFO:teuthology.orchestra.run.smithi139.stdout: "tags": [], 2021-09-06T11:25:07.416 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": [ 2021-09-06T11:25:07.416 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD" 2021-09-06T11:25:07.416 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:07.416 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:07.417 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:25:07.417 INFO:teuthology.orchestra.run.smithi139.stdout: "default_placement": "default-placement", 2021-09-06T11:25:07.417 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:25:07.417 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_policy": { 2021-09-06T11:25:07.418 INFO:teuthology.orchestra.run.smithi139.stdout: "groups": [] 2021-09-06T11:25:07.418 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:07.418 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:07.418 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:25:07.419 INFO:teuthology.orchestra.run.smithi139.stdout: "short_zone_ids": [ 2021-09-06T11:25:07.419 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:07.420 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:25:07.420 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 4134315483 2021-09-06T11:25:07.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:07.420 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:07.421 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "4487d356-c18b-4713-bd68-02dfa183d02a", 2021-09-06T11:25:07.421 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 2859952350 2021-09-06T11:25:07.421 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:07.421 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:07.422 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:25:07.422 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 772393743 2021-09-06T11:25:07.422 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:07.422 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:07.422 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T11:25:07.423 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 242578107 2021-09-06T11:25:07.423 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:07.423 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:07.423 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:07.424 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T11:25:07.424 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:25:07.424 INFO:teuthology.orchestra.run.smithi139.stdout: "period_config": { 2021-09-06T11:25:07.424 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_quota": { 2021-09-06T11:25:07.424 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2021-09-06T11:25:07.425 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2021-09-06T11:25:07.425 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2021-09-06T11:25:07.425 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2021-09-06T11:25:07.425 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2021-09-06T11:25:07.425 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:07.426 INFO:teuthology.orchestra.run.smithi139.stdout: "user_quota": { 2021-09-06T11:25:07.426 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2021-09-06T11:25:07.426 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2021-09-06T11:25:07.426 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2021-09-06T11:25:07.427 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2021-09-06T11:25:07.427 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2021-09-06T11:25:07.427 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:07.427 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:07.428 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:25:07.428 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_name": "test-realm", 2021-09-06T11:25:07.428 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:07.428 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:25:07.429 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-09-06T11:25:07.429 INFO:teuthology.orchestra.run:waiting for 300 2021-09-06T11:25:07.462 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2021-09-06T11:25:07.462+0000 7f4254a4e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 65095) UID: 0 2021-09-06T11:25:07.462 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2021-09-06T11:25:07.462+0000 7f42658ec440 -1 shutting down 2021-09-06T11:25:07.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-09-06T11:25:13.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~6s 2021-09-06T11:25:13.532 INFO:tasks.rgw.c2.client.1:Stopped 2021-09-06T11:25:13.532 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-06T11:25:13.533 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-06T11:25:13.536 INFO:tasks.rgw.c2.client.1:Started 2021-09-06T11:25:13.537 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8001 2021-09-06T11:25:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-06T11:25:13.559 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:25:13.559 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:25:13.560 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi139.front.sepia.ceph.com port 8001: Connection refused 2021-09-06T11:25:14.560 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8001 2021-09-06T11:25:14.580 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T11:25:14.580 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2021-09-06T11:25:14.581 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2021-09-06T11:25:14.582 INFO:teuthology.orchestra.run.smithi139.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-006135fa9a-10e6-test-zone4","HostId":"10e6-test-zone4-test-zonegroup"} 2021-09-06T11:25:14.582 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-09-06T11:25:14.582 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-09-06T11:25:14.596 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph-c_2227f2382f1df5117a62687acd2d99ea5eeb871e/qa/../src/test/rgw/rgw_multi 2021-09-06T11:25:14.610 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-09-06T11:25:14.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:25:14.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:25:14.612 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T11:25:14.663 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:14.663 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:14.719 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:25:14.720 INFO:teuthology.orchestra.run.smithi094.stdout: "user_id": "rgw-multisite-test-user", 2021-09-06T11:25:14.720 INFO:teuthology.orchestra.run.smithi094.stdout: "display_name": "Multisite Test User", 2021-09-06T11:25:14.720 INFO:teuthology.orchestra.run.smithi094.stdout: "email": "", 2021-09-06T11:25:14.720 INFO:teuthology.orchestra.run.smithi094.stdout: "suspended": 0, 2021-09-06T11:25:14.721 INFO:teuthology.orchestra.run.smithi094.stdout: "max_buckets": 1000, 2021-09-06T11:25:14.721 INFO:teuthology.orchestra.run.smithi094.stdout: "subusers": [], 2021-09-06T11:25:14.721 INFO:teuthology.orchestra.run.smithi094.stdout: "keys": [ 2021-09-06T11:25:14.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:25:14.722 INFO:teuthology.orchestra.run.smithi094.stdout: "user": "rgw-multisite-test-user", 2021-09-06T11:25:14.722 INFO:teuthology.orchestra.run.smithi094.stdout: "access_key": "TXRDMT4IBTDUDIML9W90", 2021-09-06T11:25:14.722 INFO:teuthology.orchestra.run.smithi094.stdout: "secret_key": "Xabfb2VbUtEGIY69tYdogKRfR9shCwewL1dEPK1Q" 2021-09-06T11:25:14.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:25:14.722 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T11:25:14.723 INFO:teuthology.orchestra.run.smithi094.stdout: "swift_keys": [], 2021-09-06T11:25:14.723 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": [], 2021-09-06T11:25:14.723 INFO:teuthology.orchestra.run.smithi094.stdout: "op_mask": "read, write, delete", 2021-09-06T11:25:14.723 INFO:teuthology.orchestra.run.smithi094.stdout: "default_placement": "", 2021-09-06T11:25:14.724 INFO:teuthology.orchestra.run.smithi094.stdout: "default_storage_class": "", 2021-09-06T11:25:14.724 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_tags": [], 2021-09-06T11:25:14.724 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_quota": { 2021-09-06T11:25:14.724 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T11:25:14.724 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T11:25:14.725 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T11:25:14.725 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T11:25:14.725 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T11:25:14.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:25:14.726 INFO:teuthology.orchestra.run.smithi094.stdout: "user_quota": { 2021-09-06T11:25:14.726 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T11:25:14.726 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T11:25:14.726 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T11:25:14.727 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T11:25:14.727 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T11:25:14.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:25:14.727 INFO:teuthology.orchestra.run.smithi094.stdout: "temp_url_keys": [], 2021-09-06T11:25:14.728 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "rgw", 2021-09-06T11:25:14.728 INFO:teuthology.orchestra.run.smithi094.stdout: "mfa_ids": [] 2021-09-06T11:25:14.728 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:25:14.728 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-09-06T11:25:14.734 INFO:rgw_multi.tests:meta checkpoint 2021-09-06T11:25:14.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:14.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:14.735 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:25:14.776 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:14.777 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:14.819 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:25:14.834 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:25:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:25:14.834 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:25:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:25:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:25:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:25:14.835 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:25:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:25:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:25:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:25:14.836 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:25:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:25:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:25:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:25:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:25:14.837 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:25:14.838 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:25:14.838 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:25:14.839 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:25:14.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:14.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:14.840 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:25:14.882 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:14.883 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:14.943 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:25:14.943 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:25:14.944 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:25:14.944 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:25:14.944 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:25:14.944 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:14.945 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:14.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:14.945 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:25:14.945 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:14.946 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:25:14.946 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:14.946 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:14.946 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:14.947 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:14.947 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:25:14.947 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:14.947 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:14.948 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:14.948 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:14.948 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:14.948 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:14.949 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:25:14.949 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:14.949 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:14.949 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:14.949 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:14.950 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:14.950 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:14.950 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:14.950 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:14.951 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:14.951 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:14.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:14.951 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:25:14.952 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:14.952 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:14.952 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:14.952 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:14.952 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:14.953 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:14.953 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:14.953 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:14.953 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:14.954 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:14.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:14.954 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:25:14.954 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:14.955 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:14.955 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:14.955 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:14.956 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:14.956 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:14.956 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:14.956 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:14.957 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:14.957 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:14.957 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:14.957 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:14.958 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:25:14.958 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:25:14.958 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:25:14.958 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:14.958 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:25:14.960 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:25:19.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-09-06T11:25:19.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:19.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:19.962 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:25:20.010 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:20.011 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:20.079 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:25:20.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:20.082 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:25:20.082 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:20.082 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:20.085 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:25:20.085 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:20.085 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:20.085 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:20.093 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:20.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:20.093 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:25:20.093 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:25:20.093 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:25:20.094 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:20.094 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:25:20.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:25:25.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:25.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:25.097 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:25:25.147 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:25.147 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:25.211 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:25:25.212 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:25:25.212 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:25:25.212 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:25:25.213 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:25:25.213 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:25.213 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:25.213 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:25.214 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:25:25.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:25.214 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:25:25.214 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:25.215 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:25.215 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:25.215 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:25.215 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:25:25.216 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:25.216 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:25.216 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:25.216 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:25.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:25.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:25.217 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:25:25.217 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:25.217 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:25.218 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:25.218 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:25.218 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:25.218 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:25.219 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:25.219 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:25.219 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:25.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:25.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:25.220 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:25:25.220 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:25.220 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:25.220 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:25.221 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:25.221 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:25.221 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:25.221 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:25.222 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:25.222 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:25.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:25.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:25.223 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:25:25.223 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:25.223 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:25.223 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:25.223 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:25.224 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:25.224 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:25.224 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:25.224 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:25.225 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:25.225 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:25.225 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:25.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:25.226 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:25:25.226 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:25:25.226 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:25:25.226 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:25.226 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:25:25.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:25:30.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:30.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:30.230 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:25:30.277 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:30.277 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:30.344 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:25:30.345 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:25:30.345 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:25:30.345 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:25:30.346 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:25:30.346 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:30.346 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:30.346 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:30.347 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:25:30.347 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:30.347 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:25:30.347 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:30.347 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:30.348 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:30.348 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:30.348 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:25:30.348 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:30.349 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:30.349 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:30.349 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:30.349 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:30.349 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:30.350 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:25:30.350 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:30.350 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:30.350 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:30.351 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:30.351 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:30.351 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:30.351 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:30.352 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:30.352 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:30.352 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:30.352 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:30.353 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:25:30.353 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:30.353 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:30.353 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:30.353 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:30.354 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:30.354 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:30.354 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:30.354 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:30.355 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:30.355 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:30.355 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:30.355 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:25:30.356 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:30.356 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:30.356 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:30.356 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:30.356 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:30.357 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:30.357 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:30.357 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:30.357 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:30.358 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:30.358 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:30.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:30.358 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:25:30.359 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:25:30.359 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:25:30.359 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:30.359 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:25:30.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:25:35.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:35.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:35.362 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:25:35.410 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:35.411 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:35.475 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:25:35.475 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:25:35.475 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:25:35.475 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:25:35.476 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:25:35.476 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:35.476 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:35.476 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:35.476 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:25:35.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:35.477 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:25:35.477 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:35.477 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:35.477 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:35.478 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:35.478 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:25:35.478 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:35.478 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:35.479 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:35.479 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:35.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:35.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:35.479 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:25:35.480 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:35.480 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:35.480 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:35.480 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:35.480 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:35.481 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:35.481 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:35.481 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:35.481 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:35.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:35.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:35.482 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:25:35.482 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:35.482 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:35.482 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:35.483 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:35.483 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:35.483 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:35.483 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:35.483 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:35.484 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:35.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:35.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:35.484 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:25:35.485 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:35.485 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:35.485 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:35.485 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:35.485 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:35.486 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:35.486 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:35.486 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:35.486 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:35.486 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:35.487 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:35.487 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:35.487 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:25:35.487 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:25:35.487 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:25:35.488 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:35.488 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:25:35.489 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:25:40.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:40.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:40.491 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:25:40.539 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:40.540 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:40.613 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:25:40.614 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:25:40.614 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:25:40.614 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:25:40.614 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:25:40.615 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:40.615 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:40.615 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:40.615 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:25:40.616 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:40.616 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:25:40.616 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:40.616 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:40.617 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:40.617 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:40.617 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:25:40.617 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:40.617 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:40.618 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:40.618 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:40.618 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:40.618 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:40.619 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:25:40.619 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:40.619 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:40.619 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:40.620 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:40.620 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:40.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:40.620 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:40.621 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:40.621 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:40.621 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:40.621 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:40.621 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:25:40.622 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:40.622 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:40.622 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:40.622 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:40.623 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:40.623 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:40.623 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:40.623 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:40.624 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:40.624 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:40.624 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:40.624 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:25:40.624 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:40.625 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:40.625 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:40.625 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:40.625 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:40.626 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:40.626 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:40.626 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:40.626 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:40.627 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:40.627 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:40.627 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:40.627 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:25:40.627 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:25:40.628 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:25:40.628 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:40.628 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:25:40.629 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:25:45.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:45.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:45.632 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:25:45.680 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:45.680 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:45.745 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:25:45.746 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:25:45.746 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:25:45.746 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:25:45.747 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:25:45.747 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:45.747 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:45.747 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:45.747 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:25:45.748 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:45.748 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:25:45.748 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:45.748 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:45.749 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:45.749 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:45.749 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:25:45.749 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:45.749 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:45.750 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:45.750 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:45.750 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:45.750 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:45.750 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:25:45.751 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:45.751 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:45.751 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:45.751 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:45.752 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:45.752 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:45.752 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:45.752 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:45.753 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:45.753 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:45.753 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:45.753 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:25:45.753 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:45.754 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:45.754 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:45.754 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:45.754 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:45.755 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:45.755 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:45.755 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:45.755 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:45.756 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:45.756 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:45.756 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:25:45.756 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:45.756 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:45.757 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:45.757 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:45.757 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:45.757 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:45.757 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:45.758 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:45.758 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:45.758 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:45.758 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:45.759 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:45.759 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:25:45.759 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:25:45.759 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:25:45.759 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:45.760 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:25:45.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:25:50.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:50.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:50.763 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:25:50.812 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:50.812 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:50.879 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:25:50.879 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:25:50.880 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:25:50.880 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:25:50.880 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:25:50.880 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:50.881 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:50.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:50.881 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:25:50.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:50.882 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:25:50.882 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:50.882 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:50.883 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:50.883 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:50.883 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:25:50.883 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:50.884 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:50.884 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:50.884 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:50.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:50.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:50.885 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:25:50.885 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:50.885 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:50.886 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:50.886 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:50.886 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:50.886 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:50.887 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:50.887 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:50.887 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:50.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:50.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:50.888 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:25:50.888 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:50.888 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:50.889 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:50.889 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:50.889 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:50.889 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:50.890 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:50.890 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:50.890 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:50.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:50.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:50.891 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:25:50.891 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:50.891 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:50.892 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:50.892 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:50.892 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:50.892 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:50.892 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:50.893 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:50.893 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:50.893 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:50.893 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:50.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:50.894 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:25:50.894 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:25:50.894 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:25:50.895 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:50.895 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:25:50.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:25:55.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:55.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:25:55.899 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:25:55.947 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:25:55.947 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:25:56.014 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:25:56.014 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:25:56.015 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:25:56.015 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:25:56.015 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:25:56.016 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:25:56.016 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:56.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:56.017 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:25:56.017 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:56.017 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:25:56.017 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:56.017 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:56.018 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:56.018 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:56.018 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:25:56.018 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:56.019 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:56.019 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:56.019 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:56.019 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:56.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:56.020 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:25:56.020 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:56.020 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:56.021 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:56.021 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:56.021 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:56.021 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:56.022 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:56.022 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:56.022 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:56.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:56.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:56.023 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:25:56.023 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:56.023 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:56.023 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:56.024 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:56.024 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:56.024 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:56.024 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:56.025 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:56.025 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:56.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:56.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:25:56.026 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:25:56.026 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:25:56.026 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:25:56.026 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:25:56.027 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:25:56.027 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:25:56.027 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:25:56.027 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:25:56.027 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:25:56.028 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:56.028 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:56.028 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:25:56.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:25:56.029 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:25:56.029 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:25:56.029 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:25:56.029 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:25:56.030 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:25:56.031 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:01.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:01.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:01.034 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:01.083 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:01.083 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:01.150 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:01.151 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:01.152 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:01.153 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:01.154 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:01.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:01.156 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:01.157 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:01.158 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:01.159 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:01.160 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:01.161 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:01.162 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:01.163 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:01.164 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:01.165 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:01.165 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:01.165 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:01.166 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:06.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:06.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:06.168 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:06.218 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:06.218 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:06.287 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:06.288 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:06.288 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:06.288 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:06.288 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:06.289 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:06.289 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:06.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:06.289 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:06.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:06.290 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:06.290 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:06.290 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:06.290 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:06.291 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:06.291 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:06.291 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:06.291 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:06.291 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:06.292 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:06.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:06.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:06.292 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:06.293 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:06.293 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:06.293 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:06.293 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:06.293 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:06.294 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:06.294 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:06.294 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:06.294 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:06.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:06.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:06.295 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:06.295 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:06.295 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:06.296 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:06.296 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:06.296 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:06.296 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:06.296 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:06.297 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:06.297 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:06.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:06.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:06.297 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:06.298 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:06.298 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:06.298 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:06.298 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:06.299 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:06.299 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:06.299 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:06.299 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:06.299 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:06.300 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:06.300 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:06.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:06.300 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:06.300 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:06.301 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:06.301 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:06.301 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:06.302 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:11.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:11.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:11.304 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:11.351 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:11.351 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:11.421 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:11.422 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:11.422 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:11.422 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:11.423 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:11.423 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:11.423 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:11.423 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:11.423 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:11.424 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:11.424 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:11.424 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:11.424 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:11.424 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:11.425 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:11.425 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:11.425 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:11.425 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:11.426 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:11.426 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:11.426 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:11.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:11.427 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:11.427 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:11.427 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:11.427 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:11.427 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:11.428 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:11.428 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:11.428 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:11.428 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:11.429 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:11.429 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:11.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:11.429 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:11.429 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:11.430 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:11.430 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:11.430 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:11.430 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:11.430 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:11.431 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:11.431 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:11.431 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:11.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:11.431 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:11.432 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:11.432 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:11.432 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:11.432 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:11.433 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:11.433 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:11.433 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:11.433 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:11.433 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:11.434 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:11.434 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:11.434 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:11.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:11.434 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:11.435 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:11.435 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:11.435 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:11.435 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:11.437 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:16.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:16.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:16.439 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:16.553 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:16.553 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:16.554 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:16.554 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:16.554 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:16.554 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:16.555 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:16.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:16.555 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:16.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:16.555 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:16.556 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:16.556 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:16.556 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:16.556 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:16.557 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:16.557 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:16.557 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:16.557 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:16.558 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:16.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:16.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:16.558 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:16.558 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:16.559 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:16.559 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:16.559 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:16.559 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:16.560 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:16.560 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:16.560 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:16.560 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:16.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:16.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:16.561 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:16.561 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:16.561 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:16.562 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:16.562 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:16.562 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:16.562 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:16.562 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:16.563 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:16.563 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:16.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:16.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:16.564 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:16.564 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:16.564 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:16.564 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:16.564 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:16.565 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:16.565 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:16.565 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:16.565 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:16.566 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:16.566 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:16.566 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:16.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:16.566 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:16.567 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:16.567 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:16.567 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:16.567 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:16.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:21.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:21.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:21.571 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:21.619 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:21.619 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:21.688 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:21.689 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:21.689 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:21.689 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:21.690 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:21.690 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:21.690 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:21.690 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:21.691 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:21.691 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:21.691 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:21.691 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:21.692 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:21.692 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:21.692 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:21.692 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:21.692 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:21.693 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:21.693 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:21.693 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:21.693 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:21.694 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:21.694 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:21.694 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:21.694 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:21.695 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:21.695 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:21.695 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:21.695 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:21.696 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:21.696 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:21.696 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:21.696 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:21.696 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:21.697 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:21.697 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:21.698 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:21.698 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:21.699 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:21.699 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:21.699 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:21.699 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:21.699 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:21.700 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:21.700 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:21.700 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:21.700 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:21.701 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:21.701 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:21.701 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:21.701 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:21.702 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:21.702 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:21.702 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:21.702 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:21.702 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:21.703 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:21.703 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:21.703 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:21.703 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:21.704 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:21.704 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:21.704 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:21.704 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:21.705 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:26.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:26.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:26.708 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:26.756 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:26.756 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:26.824 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:26.825 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:26.825 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:26.825 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:26.825 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:26.826 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:26.826 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:26.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:26.826 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:26.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:26.827 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:26.827 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:26.827 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:26.827 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:26.827 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:26.828 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:26.828 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:26.828 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:26.828 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:26.829 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:26.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:26.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:26.829 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:26.829 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:26.830 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:26.830 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:26.830 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:26.830 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:26.830 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:26.831 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:26.831 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:26.831 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:26.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:26.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:26.832 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:26.832 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:26.832 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:26.833 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:26.833 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:26.833 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:26.834 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:26.834 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:26.834 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:26.835 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:26.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:26.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:26.836 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:26.836 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:26.837 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:26.837 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:26.838 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:26.838 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:26.838 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:26.839 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:26.839 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:26.840 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:26.840 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:26.841 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:26.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:26.841 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:26.842 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:26.842 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:26.843 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:26.843 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:26.845 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:31.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:31.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:31.847 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:31.896 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:31.897 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:31.971 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:31.971 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:31.972 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:31.972 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:31.972 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:31.972 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:31.973 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:31.973 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:31.973 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:31.973 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:31.973 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:31.974 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:31.974 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:31.974 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:31.974 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:31.975 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:31.975 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:31.975 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:31.975 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:31.976 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:31.976 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:31.976 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:31.976 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:31.976 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:31.977 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:31.977 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:31.977 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:31.977 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:31.978 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:31.978 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:31.978 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:31.978 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:31.978 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:31.979 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:31.979 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:31.979 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:31.979 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:31.980 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:31.980 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:31.980 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:31.980 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:31.981 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:31.981 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:31.981 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:31.981 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:31.981 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:31.982 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:31.982 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:31.982 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:31.982 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:31.983 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:31.983 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:31.983 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:31.983 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:31.983 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:31.984 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:31.984 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:31.984 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:31.984 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:31.985 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:31.985 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:31.985 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:31.985 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:31.985 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:31.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:36.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:36.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:36.989 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:37.039 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:37.040 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:37.114 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:37.114 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:37.115 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:37.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:37.117 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:37.118 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:37.119 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:37.120 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:37.121 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:37.122 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:37.123 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:37.124 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:37.125 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:37.126 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:37.126 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:37.126 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:37.126 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:37.126 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:37.127 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:37.127 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:37.127 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:37.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:37.128 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:37.128 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:37.128 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:37.128 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:37.128 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:37.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:42.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:42.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:42.132 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:42.180 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:42.180 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:42.246 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:42.246 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:42.247 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:42.247 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:42.247 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:42.247 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:42.248 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:42.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:42.248 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:42.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:42.249 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:42.249 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:42.249 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:42.249 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:42.250 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:42.250 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:42.250 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:42.250 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:42.250 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:42.251 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:42.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:42.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:42.251 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:42.251 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:42.252 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:42.252 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:42.252 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:42.252 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:42.253 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:42.253 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:42.253 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:42.253 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:42.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:42.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:42.254 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:42.254 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:42.254 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:42.254 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:42.255 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:42.255 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:42.255 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:42.255 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:42.255 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:42.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:42.256 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:42.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:42.256 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:42.256 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:42.257 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:42.257 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:42.257 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:42.257 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:42.257 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:42.258 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:42.258 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:42.258 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:42.258 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:42.259 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:42.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:42.259 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:42.259 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:42.259 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:42.260 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:42.260 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:42.261 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:47.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:47.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:47.263 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:47.314 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:47.314 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:47.385 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:47.386 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:47.386 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:47.386 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:47.386 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:47.387 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:47.387 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:47.387 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:47.387 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:47.388 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:47.388 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:47.388 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:47.388 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:47.388 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:47.389 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:47.389 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:47.389 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:47.389 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:47.389 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:47.390 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:47.390 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:47.390 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:47.390 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:47.391 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:47.391 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:47.391 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:47.391 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:47.391 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:47.392 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:47.392 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:47.392 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:47.392 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:47.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:47.393 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:47.393 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:47.393 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:47.393 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:47.393 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:47.394 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:47.394 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:47.394 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:47.394 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:47.395 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:47.395 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:47.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:47.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:47.395 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:47.396 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:47.396 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:47.396 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:47.396 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:47.396 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:47.397 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:47.397 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:47.397 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:47.397 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:47.397 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:47.398 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:47.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:47.398 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:47.398 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:47.398 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:47.399 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:47.399 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:47.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:52.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:52.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:52.403 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:52.451 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:52.451 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:52.521 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:52.521 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:52.521 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:52.522 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:52.522 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:52.522 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:52.523 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:52.523 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:52.523 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:52.523 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:52.523 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:52.524 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:52.524 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:52.524 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:52.524 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:52.525 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:52.525 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:52.525 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:52.526 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:52.526 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:52.526 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:52.526 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:52.527 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:52.527 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:52.527 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:52.527 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:52.528 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:52.528 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:52.528 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:52.528 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:52.529 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:52.529 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:52.529 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:52.529 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:52.530 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:52.530 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:52.530 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:52.530 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:52.531 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:52.531 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:52.531 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:52.531 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:52.532 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:52.532 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:52.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:52.532 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:52.533 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:52.533 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:52.533 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:52.533 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:52.534 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:52.534 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:52.534 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:52.534 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:52.535 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:52.535 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:52.535 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:52.535 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:52.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:52.536 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:52.536 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:52.536 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:52.537 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:52.537 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:52.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:26:57.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:57.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:26:57.541 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:26:57.590 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:26:57.591 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:26:57.658 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:26:57.658 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:26:57.658 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:26:57.659 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:26:57.659 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:26:57.659 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:26:57.659 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:57.659 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:57.660 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:26:57.660 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:57.660 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:26:57.660 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:57.660 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:57.661 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:57.661 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:57.661 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:26:57.661 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:57.662 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:57.662 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:57.662 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:57.662 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:57.663 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:57.663 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:26:57.663 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:57.663 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:57.663 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:57.664 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:57.664 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:57.664 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:57.664 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:57.664 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:57.665 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:57.665 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:57.665 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:57.665 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:26:57.666 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:57.666 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:57.666 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:57.666 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:57.666 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:57.667 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:57.667 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:57.667 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:57.667 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:57.667 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:57.668 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:26:57.668 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:26:57.668 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:26:57.668 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:26:57.669 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:26:57.669 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:26:57.669 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:26:57.669 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:26:57.669 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:26:57.670 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:26:57.670 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:57.670 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:57.670 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:26:57.670 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:26:57.671 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:26:57.671 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:26:57.671 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:26:57.671 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:26:57.672 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:26:57.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:27:02.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:02.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:02.675 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:02.723 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:02.723 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:02.796 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:02.797 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:02.797 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:02.797 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:02.797 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:02.798 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:02.798 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:02.798 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:02.798 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:02.799 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:02.799 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:02.799 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:02.799 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:02.799 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:02.800 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:02.800 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:02.800 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:02.800 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:02.801 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:02.801 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:02.801 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:02.801 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:02.802 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:02.802 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:02.802 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:02.802 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:02.803 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:02.803 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:02.803 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:02.803 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:02.803 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:02.804 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:02.804 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:02.804 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:02.804 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:02.805 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:02.805 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:02.805 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:02.805 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:02.805 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:02.806 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:02.806 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:02.806 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:02.806 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:02.807 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:02.807 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:02.807 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:02.807 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:02.807 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:02.808 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:02.808 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:02.808 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:02.808 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:02.808 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:02.809 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:02.809 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:02.809 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:02.810 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:02.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:02.810 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:02.810 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:02.811 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:02.811 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:02.811 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:02.812 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:27:07.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:07.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:07.815 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:07.863 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:07.863 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:07.929 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:07.929 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:07.930 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:07.930 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:07.930 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:07.930 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:07.930 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:07.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:07.931 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:07.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:07.931 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:07.931 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:07.932 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:07.932 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:07.932 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:07.932 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:07.933 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:07.933 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:07.933 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:07.933 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:07.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:07.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:07.934 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:07.934 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:07.934 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:07.934 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:07.935 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:07.935 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:07.935 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:07.935 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:07.936 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:07.936 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:07.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:07.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:07.936 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:07.937 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:07.937 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:07.937 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:07.937 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:07.937 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:07.938 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:07.938 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:07.938 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:07.938 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:07.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:07.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:07.939 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:07.939 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:07.940 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:07.940 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:07.940 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:07.940 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:07.940 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:07.941 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:07.941 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:07.941 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:07.941 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:07.941 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:07.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:07.942 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:07.942 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:07.942 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:07.942 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:07.943 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:07.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630927514.716931_25.1 target= 2021-09-06T11:27:12.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:12.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:12.946 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:12.994 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:12.994 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:13.068 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:13.068 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:13.068 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:13.069 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:13.069 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:13.069 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:13.069 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.070 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:13.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.070 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:13.070 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.071 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.071 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:13.071 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.071 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:13.072 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.072 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:13.072 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.072 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.072 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.073 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.073 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:13.073 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.073 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.073 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.074 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.074 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.074 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.074 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.075 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.075 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.076 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:13.076 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.076 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.076 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.076 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.077 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.077 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.077 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.077 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.077 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.078 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:13.078 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.079 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.079 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.079 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.079 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.079 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.080 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.080 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.080 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.080 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.081 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:13.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.081 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:13.081 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:13.081 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:13.082 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.082 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:13.083 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:27:13.084 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:27:13.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.085 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:13.133 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:13.133 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:13.199 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:27:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:27:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:27:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:27:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:27:13.200 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:27:13.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:27:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:13.202 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:27:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:13.203 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.204 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:27:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:13.205 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:13.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:27:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:13.207 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:13.208 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:27:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:13.210 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.211 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:27:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.212 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:27:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:27:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:27:13.213 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.213 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:27:13.214 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:27:13.215 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:27:13.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.216 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:13.260 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:13.260 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:13.319 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:13.319 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:13.319 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:13.319 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:13.320 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:13.320 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:13.320 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.320 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:13.321 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.321 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:13.321 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.321 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.322 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:13.322 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.322 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:13.322 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.323 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:13.323 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.323 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.324 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:13.324 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.324 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.324 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.325 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.325 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.325 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.325 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.326 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.326 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.326 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:13.327 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.327 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.327 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.327 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.328 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.328 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.328 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.328 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.328 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.329 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.329 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:13.329 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.330 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.330 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.330 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.330 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.331 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.331 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.331 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.331 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.331 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.332 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:13.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.332 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:13.332 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:13.332 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:13.333 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.333 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:13.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:27:13.334 INFO:rgw_multi.tests:meta checkpoint 2021-09-06T11:27:13.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.335 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:13.377 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:13.377 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:13.421 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:27:13.435 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:27:13.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.436 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:13.436 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:27:13.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.437 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:13.437 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:13.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.438 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:13.438 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:13.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.439 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:13.439 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:13.439 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.439 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:27:13.440 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:27:13.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.442 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:13.483 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:13.483 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:13.545 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:13.546 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:13.546 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:13.546 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:13.547 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:13.547 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:13.547 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.547 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:13.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.548 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:13.548 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.548 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.549 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:13.549 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.549 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:13.549 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.550 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:13.550 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.550 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.551 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.551 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:13.551 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.551 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.551 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.552 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.552 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.552 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.552 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.553 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.553 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.553 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:13.554 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.554 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.554 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.554 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.555 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.555 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.555 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.555 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.556 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.556 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.556 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.556 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:13.556 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.557 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.557 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.557 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.557 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.557 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.558 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.558 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.558 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.558 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.559 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:13.559 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.559 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:13.559 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:13.559 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:13.560 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.560 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:13.561 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:27:13.561 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:27:13.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.563 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:13.604 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:13.605 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:13.666 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:27:13.666 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:27:13.666 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:27:13.667 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:27:13.667 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:27:13.667 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:13.667 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:13.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.668 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:27:13.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.668 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:27:13.668 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:13.669 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:13.669 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:13.669 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:13.669 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:27:13.670 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:13.670 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:13.670 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:13.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.671 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:27:13.671 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:13.672 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:13.672 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:13.672 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:13.672 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:13.672 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:13.673 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.673 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:13.673 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.674 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:27:13.674 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:13.674 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:13.674 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:13.675 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:13.675 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:13.675 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:13.675 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.676 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:13.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:13.677 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:27:13.677 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:13.677 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:13.677 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:13.677 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:13.678 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:13.678 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:13.678 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.678 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:13.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.679 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:27:13.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:13.680 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:27:13.680 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:27:13.680 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:27:13.680 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:13.680 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:27:13.682 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:27:13.682 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:27:13.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:13.683 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:13.726 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:13.726 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:13.796 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:13.797 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:13.798 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:13.798 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:13.798 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:13.798 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:13.799 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.799 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.800 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:13.800 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.800 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:13.800 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.800 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.801 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:13.801 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.801 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:13.801 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.802 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:13.802 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.802 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.802 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.803 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.803 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:13.803 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.804 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.804 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.804 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.804 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.805 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.805 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.805 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.805 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.806 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.806 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:13.806 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.807 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.807 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.807 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.808 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.808 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.808 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.808 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.809 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.809 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.809 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:13.809 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:13.809 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:13.810 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:13.810 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:13.810 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:13.811 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:13.811 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:13.811 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:13.811 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:13.812 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.812 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.812 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:13.812 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:13.813 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:13.813 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:13.813 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:13.813 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:13.814 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:13.815 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:27:13.815 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph-c_2227f2382f1df5117a62687acd2d99ea5eeb871e/qa/../src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2021-09-06T11:27:13.850 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-1 2021-09-06T11:27:17.578 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-2 2021-09-06T11:27:21.271 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-3 2021-09-06T11:27:21.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:21.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:21.310 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:21.356 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:21.357 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:21.406 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:27:21.421 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:27:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:21.421 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:27:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:21.422 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:21.423 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927641.295519_58.1", 2021-09-06T11:27:21.424 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:21.295519Z" 2021-09-06T11:27:21.425 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:21.425 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:27:21.426 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:27:21.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:21.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:21.428 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:21.470 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:21.471 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:21.540 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:21.540 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:21.541 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:21.541 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:21.541 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:21.542 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:21.542 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:21.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:21.543 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:21.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:21.543 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:21.543 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:21.544 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:21.544 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:21.544 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:21.544 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:21.544 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:21.545 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:21.545 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:21.545 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:21.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:21.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:21.546 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:21.546 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:21.546 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:21.546 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:21.547 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:21.547 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:21.547 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:21.547 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:21.548 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:21.548 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:21.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:21.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:21.549 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:21.549 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:21.549 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:21.549 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:21.549 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:21.550 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:21.550 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:21.550 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:21.550 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:21.551 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:21.551 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:21.551 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:21.551 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:21.551 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:21.552 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:21.552 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927637.565509_46.1", 2021-09-06T11:27:21.552 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:21.552 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:21.552 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:21.553 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:17.565509Z", 2021-09-06T11:27:21.553 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:21.553 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:21.553 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:21.554 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:21.554 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:21.554 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:21.555 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:21.555 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:21.555 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:21.555 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:21.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630927641.295519_58.1 target=1_1630927637.565509_46.1 2021-09-06T11:27:26.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:26.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:26.560 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:26.608 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:26.609 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:26.676 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:26.677 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:26.677 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:26.677 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:26.678 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:26.678 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:26.678 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:26.678 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:26.679 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:26.679 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:26.679 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:26.679 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:26.679 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:26.680 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:26.680 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:26.680 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:26.680 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:26.681 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:26.681 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:26.681 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.681 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:26.682 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:26.682 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:26.682 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:26.682 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:26.683 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:26.683 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:26.683 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:26.683 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:26.683 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:26.684 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:26.684 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.684 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:26.684 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:26.685 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:26.685 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:26.685 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:26.685 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:26.685 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:26.686 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:26.686 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:26.686 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:26.686 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:26.687 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.687 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:26.687 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:26.687 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:26.687 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:26.688 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:26.688 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927641.295519_58.1", 2021-09-06T11:27:26.688 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:26.688 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:26.689 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:26.689 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:21.295519Z", 2021-09-06T11:27:26.689 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:26.690 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.690 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.690 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:26.690 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:26.690 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:26.691 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:26.691 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:26.691 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.691 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:26.693 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:27:26.693 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:27:26.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:26.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:26.695 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:26.743 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:26.743 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:26.808 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:27:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:27:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:27:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:27:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:27:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:27:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:27:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:27:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:27:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:26.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:26.817 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:27:26.817 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:26.817 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:26.818 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:26.818 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:26.818 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:26.818 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:26.819 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:26.819 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:26.819 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:26.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:26.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:26.820 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:27:26.820 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:26.820 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:26.820 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927641.295519_58.1", 2021-09-06T11:27:26.821 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:26.821 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:26.821 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:26.821 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:21.295519Z", 2021-09-06T11:27:26.822 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:26.822 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:26.822 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:26.822 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:27:26.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:26.823 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:27:26.823 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:27:26.823 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:27:26.824 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:26.824 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:27:26.825 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:27:26.826 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:27:26.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:26.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:26.828 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:26.871 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:26.872 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:26.933 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:26.933 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:26.934 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:26.934 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:26.935 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:26.935 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:26.936 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:26.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:26.936 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:26.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:26.937 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:26.938 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:26.938 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:26.939 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:26.939 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:26.939 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:26.940 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:26.940 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:26.941 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:26.941 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:26.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:26.942 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:26.943 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:26.943 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:26.944 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:26.944 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:26.945 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:26.945 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:26.945 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:26.946 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:26.946 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.947 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:26.947 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:26.948 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:26.948 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:26.948 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:26.949 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:26.949 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:26.950 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:26.950 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:26.950 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:26.951 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:26.951 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:26.952 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:26.953 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:26.953 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:26.953 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:26.954 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927641.295519_58.1", 2021-09-06T11:27:26.954 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:26.955 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:26.955 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:26.956 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:21.295519Z", 2021-09-06T11:27:26.956 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:26.956 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.957 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.957 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:26.958 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:26.958 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:26.958 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:26.959 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:26.959 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:26.960 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:26.962 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:27:26.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2021-09-06T11:27:26.990 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-4 2021-09-06T11:27:27.008 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-5 2021-09-06T11:27:27.037 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-6 2021-09-06T11:27:27.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:27.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:27.068 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:27.110 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:27.110 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:27.153 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:27:27.167 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:27:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:27:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927647.054563_70.1", 2021-09-06T11:27:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:27.054563Z" 2021-09-06T11:27:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:27.173 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:27:27.174 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:27:27.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:27.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:27.176 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:27.222 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:27.222 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:27.294 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:27.294 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:27.295 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:27.295 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:27.295 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:27.295 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:27.296 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:27.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:27.296 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:27.296 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:27.297 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:27.297 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:27.297 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:27.297 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:27.297 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:27.298 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:27.298 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:27.298 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:27.298 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:27.299 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:27.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:27.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:27.299 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:27.300 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:27.300 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:27.300 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:27.300 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:27.300 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:27.301 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:27.301 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:27.301 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:27.301 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:27.302 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:27.302 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:27.302 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:27.302 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:27.303 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:27.303 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:27.303 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:27.303 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:27.303 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:27.304 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:27.304 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:27.304 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:27.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:27.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:27.305 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:27.305 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:27.305 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:27.305 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927646.998907_59.1", 2021-09-06T11:27:27.306 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:27.306 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:27.306 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:27.306 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:26.998907Z", 2021-09-06T11:27:27.307 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:27.307 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:27.307 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:27.307 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:27.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:27.308 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:27.308 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:27.308 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:27.308 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:27.309 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:27.310 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630927647.054563_70.1 target=1_1630927646.998907_59.1 2021-09-06T11:27:32.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:32.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:32.313 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:32.362 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:32.363 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:32.427 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:32.428 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:32.428 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:32.428 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:32.428 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:32.429 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:32.429 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:32.429 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:32.429 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:32.430 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:32.430 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:32.430 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:32.430 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:32.430 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:32.431 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:32.431 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:32.431 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:32.431 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:32.432 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:32.432 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.432 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:32.432 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:32.433 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:32.433 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:32.433 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:32.433 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:32.433 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:32.434 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:32.434 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:32.434 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:32.434 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:32.435 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:32.435 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:32.435 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:32.436 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:32.436 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:32.436 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:32.436 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:32.436 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:32.437 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:32.437 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:32.437 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:32.437 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.438 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:32.438 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:32.438 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:32.438 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:32.439 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:32.439 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927647.054563_70.1", 2021-09-06T11:27:32.439 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:32.439 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:32.440 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:32.440 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:27.054563Z", 2021-09-06T11:27:32.440 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:32.440 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.440 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.441 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:32.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:32.441 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:32.441 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:32.441 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:32.442 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.442 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:32.443 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:27:32.444 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:27:32.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:32.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:32.445 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:32.494 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:32.494 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:32.562 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:27:32.562 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:27:32.563 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:27:32.563 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:27:32.563 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:27:32.563 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:32.564 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:32.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:32.564 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:27:32.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:32.564 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:27:32.565 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:32.565 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:32.565 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:32.565 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:32.566 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:27:32.566 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:32.566 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:32.566 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:32.567 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:32.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:32.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:32.567 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:27:32.567 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:32.568 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:32.568 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:32.568 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:32.568 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:32.569 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:32.569 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:32.569 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:32.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:32.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:32.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:32.570 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:27:32.570 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:32.570 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:32.571 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:32.571 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:32.571 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:32.571 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:32.572 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:32.572 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:32.572 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:32.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:32.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:32.573 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:27:32.573 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:32.573 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:32.573 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927647.054563_70.1", 2021-09-06T11:27:32.573 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:32.574 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:32.574 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:32.574 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:27.054563Z", 2021-09-06T11:27:32.574 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:32.575 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:32.575 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:32.575 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:27:32.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:32.575 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:27:32.576 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:27:32.576 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:27:32.576 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:32.576 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:27:32.578 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:27:32.578 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:27:32.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:32.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:32.580 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:32.622 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:32.623 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:32.681 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:32.682 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:32.682 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:32.682 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:32.682 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:32.683 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:32.683 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:32.683 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:32.683 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:32.684 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:32.684 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:32.684 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:32.684 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:32.684 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:32.685 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:32.685 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:32.685 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:32.685 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:32.686 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:32.686 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.686 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:32.686 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:32.687 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:32.687 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:32.687 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:32.687 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:32.687 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:32.688 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:32.688 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:32.688 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:32.688 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:32.689 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.689 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:32.689 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:32.689 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:32.690 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:32.690 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:32.690 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:32.690 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:32.690 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:32.691 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:32.691 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:32.691 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:32.691 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.692 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:32.692 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:32.692 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:32.692 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:32.692 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:32.693 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927647.054563_70.1", 2021-09-06T11:27:32.693 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:32.693 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:32.693 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:32.694 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:27.054563Z", 2021-09-06T11:27:32.694 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:32.694 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.694 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.694 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:32.695 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:32.695 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:32.695 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:32.695 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:32.696 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:32.696 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:32.697 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:27:32.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:32.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:32.949 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:32.991 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:32.991 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:27:33.047 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:27:33.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.048 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:33.048 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:27:33.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.049 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:33.049 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:33.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.050 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:33.050 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:33.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.051 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927647.054563_70.1", 2021-09-06T11:27:33.051 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:27.054563Z" 2021-09-06T11:27:33.051 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:33.051 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:27:33.052 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:27:33.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:33.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:33.054 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:33.097 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:33.097 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:33.170 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:33.170 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:33.171 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:33.171 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:33.171 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:33.172 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:33.172 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.172 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:33.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.173 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:33.173 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.173 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.173 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:33.174 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.174 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:33.175 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.175 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:33.176 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.176 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.177 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:33.178 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.178 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.179 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:33.179 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.180 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:33.180 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.180 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:33.181 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.181 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.183 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:33.183 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.184 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.184 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:33.186 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.186 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:33.187 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.187 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:33.188 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.188 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.190 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:33.190 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.191 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.191 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927647.054563_70.1", 2021-09-06T11:27:33.191 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.192 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:33.192 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.193 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:27.054563Z", 2021-09-06T11:27:33.193 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.194 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.194 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.195 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:33.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.196 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:33.196 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:33.196 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:33.197 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.197 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:33.200 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:27:33.200 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:27:33.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:33.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:33.203 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:33.245 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:33.246 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:33.303 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:27:33.304 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:27:33.304 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:27:33.304 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:27:33.304 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:27:33.305 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:33.305 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:33.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.305 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:27:33.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.306 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:27:33.306 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:33.306 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:33.307 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:33.307 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:33.307 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:27:33.307 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:33.308 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:33.308 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:33.308 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:33.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.309 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:27:33.309 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:33.309 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:33.310 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:33.310 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:33.310 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:33.310 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:33.311 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:33.311 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:33.311 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:33.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.312 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:27:33.312 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:33.312 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:33.312 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:33.313 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:33.313 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:33.313 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:33.313 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:33.313 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:33.314 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:33.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.314 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:27:33.315 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:33.315 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:33.315 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927647.054563_70.1", 2021-09-06T11:27:33.315 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:33.316 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:33.316 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:33.316 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:27.054563Z", 2021-09-06T11:27:33.316 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:33.316 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:33.317 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:33.317 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:27:33.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.317 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:27:33.317 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:27:33.318 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:27:33.318 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:33.318 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:27:33.319 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:27:33.320 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:27:33.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:33.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:33.321 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:33.363 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:33.363 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:33.422 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:33.423 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:33.423 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:33.423 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:33.424 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:33.424 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:33.424 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.424 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.425 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:33.425 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.425 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:33.425 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.425 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.426 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:33.426 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.426 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:33.426 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.427 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:33.427 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.427 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.427 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.427 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.428 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:33.428 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.428 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.428 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:33.429 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.429 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:33.429 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.429 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:33.429 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.430 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.430 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.430 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.430 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:33.430 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.431 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.431 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:33.431 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.431 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:33.432 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.432 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:33.432 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.432 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.432 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.433 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.433 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:33.433 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.433 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.434 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927647.054563_70.1", 2021-09-06T11:27:33.434 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.434 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:33.434 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.434 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:27.054563Z", 2021-09-06T11:27:33.435 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.435 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.435 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.435 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:33.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.436 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:33.436 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:33.436 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:33.436 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.436 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:33.437 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:27:33.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2021-09-06T11:27:33.461 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-7 2021-09-06T11:27:33.477 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-8 2021-09-06T11:27:33.504 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-9 2021-09-06T11:27:33.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:33.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:33.539 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:33.580 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:33.580 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:33.628 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:27:33.643 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:27:33.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.643 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:33.644 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:27:33.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.644 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:33.645 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:33.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.645 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:33.645 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:33.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:33.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:33.646 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927653.521775_82.1", 2021-09-06T11:27:33.646 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:33.521775Z" 2021-09-06T11:27:33.647 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:33.647 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:27:33.648 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:27:33.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:33.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:33.650 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:33.692 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:33.692 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:33.753 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:33.753 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:33.754 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:33.754 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:33.754 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:33.754 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:33.757 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.757 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:33.757 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.757 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:33.758 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.758 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.758 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.758 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.759 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:33.759 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.759 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.759 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:33.760 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.760 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:33.760 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.760 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:33.760 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.761 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.761 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.761 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.761 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:33.762 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.762 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.762 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:33.762 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.763 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:33.763 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.763 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:33.763 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.764 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.764 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.764 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:33.764 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:33.764 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:33.765 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:33.765 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927653.468540_71.1", 2021-09-06T11:27:33.765 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:33.765 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:33.766 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:33.766 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:33.468540Z", 2021-09-06T11:27:33.766 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:33.766 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.767 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.767 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:33.767 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:33.767 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:33.767 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:33.768 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:33.768 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:33.768 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:33.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630927653.521775_82.1 target=1_1630927653.468540_71.1 2021-09-06T11:27:38.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:38.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:38.772 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:38.822 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:38.822 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:38.890 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:38.891 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:38.891 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:38.892 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:38.892 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:38.893 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:38.893 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:38.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:38.894 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:38.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:38.895 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:38.895 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:38.896 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:38.896 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:38.897 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:38.897 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:38.897 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:38.898 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:38.898 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:38.899 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:38.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:38.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:38.900 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:38.901 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:38.901 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:38.901 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:38.902 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:38.902 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:38.903 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:38.903 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:38.904 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:38.904 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:38.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:38.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:38.905 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:38.906 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:38.906 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:38.907 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:38.907 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:38.908 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:38.908 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:38.909 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:38.909 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:38.909 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:38.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:38.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:38.911 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:38.911 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:38.912 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:38.912 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927653.521775_82.1", 2021-09-06T11:27:38.912 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:38.913 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:38.913 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:38.914 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:33.521775Z", 2021-09-06T11:27:38.914 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:38.915 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:38.915 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:38.916 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:38.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:38.916 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:38.917 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:38.917 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:38.918 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:38.918 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:38.920 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:27:38.921 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:27:38.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:38.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:38.924 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:38.975 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:38.975 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:39.041 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:27:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:27:39.041 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:27:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:27:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:27:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:39.042 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:39.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:39.043 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:27:39.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:39.043 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:27:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:39.044 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:27:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:39.045 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:27:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:39.047 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:39.048 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:27:39.049 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:39.050 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:39.051 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:27:39.052 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927653.521775_82.1", 2021-09-06T11:27:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:39.053 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:33.521775Z", 2021-09-06T11:27:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:27:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:27:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:27:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:27:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:39.056 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:27:39.057 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:27:39.058 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:27:39.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:39.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:39.060 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:39.105 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:39.105 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:39.176 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:39.176 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:39.177 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:39.177 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:39.177 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:39.177 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:39.177 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:39.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:39.178 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:39.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:39.178 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:39.179 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:39.179 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:39.179 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:39.179 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:39.180 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:39.180 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:39.180 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:39.180 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:39.180 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:39.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:39.181 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:39.181 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:39.182 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:39.182 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:39.182 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:39.182 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:39.183 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:39.183 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:39.183 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:39.183 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:39.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:39.184 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:39.184 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:39.184 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:39.185 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:39.185 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:39.185 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:39.185 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:39.185 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:39.186 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:39.186 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:39.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:39.187 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:39.187 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:39.187 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:39.187 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927653.521775_82.1", 2021-09-06T11:27:39.187 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:39.188 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:39.188 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:39.188 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:33.521775Z", 2021-09-06T11:27:39.189 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:39.189 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.189 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.189 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:39.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:39.190 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:39.190 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:39.190 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:39.191 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.191 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:39.192 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:27:39.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:39.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:39.363 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:39.453 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:27:39.467 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:27:39.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:27:39.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:39.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:39.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:27:39.471 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:27:39.472 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:39.472 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:27:39.473 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:27:39.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:39.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:39.475 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:39.517 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:39.518 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:39.577 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:39.578 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:39.578 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:39.578 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:39.578 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:39.579 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:39.579 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:39.579 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:39.579 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:39.579 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:39.580 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:39.580 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:39.580 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:39.580 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:39.581 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:39.581 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:39.581 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:39.581 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:39.581 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:39.582 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.582 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:39.582 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:39.582 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:39.583 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:39.583 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:39.583 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:39.583 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:39.584 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:39.584 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:39.584 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:39.584 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:39.585 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.585 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:39.585 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:39.585 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:39.585 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:39.586 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:39.586 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:39.586 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:39.586 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:39.587 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:39.587 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:39.587 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:39.587 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:39.588 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:39.588 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:39.588 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:39.588 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:39.589 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.238278_83.1", 2021-09-06T11:27:39.589 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:39.589 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:39.589 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:39.590 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.238278Z", 2021-09-06T11:27:39.590 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:39.590 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.590 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.591 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:39.591 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:39.591 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:39.591 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:39.591 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:39.592 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:39.592 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:39.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630927659.337096_88.1 target=1_1630927659.238278_83.1 2021-09-06T11:27:44.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:44.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:44.595 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:44.647 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:44.647 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:44.716 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:44.716 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:44.717 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:44.717 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:44.717 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:44.717 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:44.717 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:44.718 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:44.718 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:44.718 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:44.718 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:44.719 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:44.719 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:44.719 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:44.719 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:44.719 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:44.720 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:44.720 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:44.720 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:44.720 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.720 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:44.721 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:44.721 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:44.721 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:44.721 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:44.721 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:44.722 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:44.722 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:44.722 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:44.722 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:44.723 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:44.723 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.723 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:44.723 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:44.723 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:44.724 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:44.724 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:44.724 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:44.724 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:44.725 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:44.725 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:44.725 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:44.725 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:44.725 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.726 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:44.726 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:44.726 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:44.726 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:44.727 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:44.727 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:27:44.727 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:44.727 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:44.727 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:44.728 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:27:44.728 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:44.728 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.728 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.729 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:44.729 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:44.729 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:44.729 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:44.729 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:44.730 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.730 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:44.731 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:27:44.732 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:27:44.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:44.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:44.733 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:44.781 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:44.782 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:44.845 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:27:44.846 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:27:44.846 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:27:44.846 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:27:44.847 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:27:44.847 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:44.847 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:44.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:44.847 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:27:44.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:44.848 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:27:44.848 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:44.849 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:44.849 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:44.849 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:44.849 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:27:44.850 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:44.850 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:44.850 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:44.850 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:44.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:44.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:44.851 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:27:44.851 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:44.851 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:44.852 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:44.852 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:44.852 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:44.852 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:44.852 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:44.853 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:44.853 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:44.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:44.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:44.854 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:27:44.854 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:44.854 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:44.854 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:44.855 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:44.855 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:44.855 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:44.856 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:44.856 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:44.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:44.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:44.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:44.857 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:27:44.857 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:44.857 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:44.858 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:27:44.858 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:44.858 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:44.858 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:44.859 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:27:44.859 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:44.859 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:44.859 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:44.859 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:27:44.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:44.860 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:27:44.860 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:27:44.860 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:27:44.861 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:44.861 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:27:44.862 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:27:44.863 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:27:44.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:44.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:44.864 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:44.908 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:44.909 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:44.975 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:44.975 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:44.976 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:44.976 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:44.976 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:44.976 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:44.977 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:44.977 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:44.977 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:44.977 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:44.977 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:44.978 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:44.978 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:44.978 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:44.978 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:44.979 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:44.979 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:44.979 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:44.979 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:44.979 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:44.980 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:44.980 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:44.980 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:44.981 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:44.981 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:44.981 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:44.981 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:44.981 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:44.982 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:44.982 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:44.982 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.982 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:44.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:44.983 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:44.983 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:44.983 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:44.983 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:44.984 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:44.984 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:44.984 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:44.984 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:44.985 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:44.985 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.985 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:44.985 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:44.985 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:44.986 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:44.986 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:44.986 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:27:44.986 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:44.987 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:44.987 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:44.987 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:27:44.987 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:44.988 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.988 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.988 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:44.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:44.988 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:44.989 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:44.989 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:44.989 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:44.989 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:44.991 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:27:45.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2021-09-06T11:27:45.002 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-10 2021-09-06T11:27:45.020 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-11 2021-09-06T11:27:45.052 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-12 2021-09-06T11:27:45.202 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2021-09-06T11:27:45.202+0000 7f1431e8b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:45.203 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2021-09-06T11:27:45.202+0000 7f142d682700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:45.203 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2021-09-06T11:27:45.202+0000 7f142fe87700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:45.204 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2021-09-06T11:27:45.203+0000 7f3e4d2ac700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:45.204 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2021-09-06T11:27:45.203+0000 7f3e512b4700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:45.204 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2021-09-06T11:27:45.203+0000 7f3e4e2ae700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:45.205 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:45.203+0000 7fc4c1229700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:45.205 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:45.203+0000 7fc4c9a3a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:45.205 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:45.203+0000 7fc4c222b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:45.206 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2021-09-06T11:27:45.206+0000 7f1426e75700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:45.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:45.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:45.648 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:45.872 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:27:45.887 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:27:45.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:45.888 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:45.888 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:27:45.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:45.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:45.890 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:45.890 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:27:45.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:45.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:45.891 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927665.068925_56.1", 2021-09-06T11:27:45.892 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:45.068925Z" 2021-09-06T11:27:45.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:45.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:45.893 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:27:45.894 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:27:45.894 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:45.894 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:27:45.896 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:27:45.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:45.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:45.899 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:45.948 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:45.949 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:46.020 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:46.021 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:46.021 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:46.021 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:46.021 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:46.022 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:46.022 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:46.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:46.022 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:46.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:46.023 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:46.023 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:46.023 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:46.023 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:46.024 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:46.024 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:46.024 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:46.025 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:46.025 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:46.025 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:46.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:46.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:46.026 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:46.026 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:46.026 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:46.026 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:46.027 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:46.027 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:46.027 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:46.027 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:46.028 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:46.028 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:46.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:46.028 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:46.029 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:46.029 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:46.029 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:46.029 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927665.010817_45.1", 2021-09-06T11:27:46.029 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:46.030 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:46.030 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:46.030 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:45.010817Z", 2021-09-06T11:27:46.030 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:46.031 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:46.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:46.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:46.032 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:46.032 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:46.032 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:46.032 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:27:46.032 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:46.033 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:46.033 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:46.033 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:27:46.033 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:46.034 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:46.034 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:46.034 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:46.034 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:46.034 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:46.035 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:46.035 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:46.035 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:46.035 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:46.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630927665.068925_56.1 target=1_1630927665.010817_45.1 2021-09-06T11:27:46.381 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:46.382+0000 7fc4c222b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:46.382 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:46.382+0000 7fc4bca20700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:46.382 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:46.382+0000 7fc4cba3e700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:46.383 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:46.384+0000 7fc4c8a38700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:46.602 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:46.602+0000 7fc4be223700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:46.602 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:46.603+0000 7fc4c422f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:46.603 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:46.603+0000 7fc4c422f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:46.608 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:27:46.607+0000 7fc4c7a36700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:27:51.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:51.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:51.039 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:51.085 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:51.086 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:51.149 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:51.150 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:51.150 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:51.150 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:51.151 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:51.151 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:51.151 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:51.151 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:51.152 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:51.152 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:51.152 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:51.152 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:51.153 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:51.153 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:51.153 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:51.153 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:51.153 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:51.154 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:51.154 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:51.154 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:51.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:51.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:51.155 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:51.155 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:51.155 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:51.156 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:51.156 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:51.156 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:51.156 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:51.157 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:51.157 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:51.157 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:51.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:51.158 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:51.158 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:51.158 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:51.158 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:51.158 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927665.068925_56.1", 2021-09-06T11:27:51.159 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:51.159 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:51.159 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:51.159 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:45.068925Z", 2021-09-06T11:27:51.160 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:51.160 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:51.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:51.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:51.160 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:51.161 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:51.161 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:51.161 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:27:51.161 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:51.162 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:51.162 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:51.162 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:27:51.162 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:51.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:51.163 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:51.163 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:51.163 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:51.163 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:51.164 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:51.164 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:51.164 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:51.164 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:51.165 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:27:51.166 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:27:51.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:51.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:51.168 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:51.213 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:51.213 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:51.274 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:27:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:27:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:27:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:27:51.275 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:27:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:51.276 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:27:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:27:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:51.277 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:27:51.278 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:27:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:51.280 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:51.281 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:51.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:27:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927665.065861_55.1", 2021-09-06T11:27:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:51.284 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:51.284 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:51.284 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:45.065861Z", 2021-09-06T11:27:51.284 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:51.285 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:51.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:51.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:51.285 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:27:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:27:51.286 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:27:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:51.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:27:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:51.288 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:27:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:27:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:27:51.289 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:51.289 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:27:51.290 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1630927665.068925_56.1 target=1_1630927665.065861_55.1 2021-09-06T11:27:56.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:56.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:56.293 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:56.341 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:56.342 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:56.405 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:27:56.405 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:27:56.406 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:27:56.406 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:27:56.406 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:27:56.406 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:56.407 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:56.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:56.407 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:27:56.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:56.408 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:27:56.408 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:56.408 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:56.408 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:56.408 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:56.409 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:27:56.409 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:56.409 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:56.409 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:56.410 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:56.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:56.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:56.410 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:27:56.410 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:56.411 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:56.411 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:27:56.411 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:56.411 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:56.412 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:56.412 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:27:56.412 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:56.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:56.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:56.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:56.413 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:27:56.413 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:56.413 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:56.413 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927665.068925_56.1", 2021-09-06T11:27:56.414 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:56.414 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:56.414 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:56.414 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:45.068925Z", 2021-09-06T11:27:56.414 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:56.415 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:56.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:56.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:27:56.415 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:27:56.416 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:27:56.416 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:27:56.416 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:27:56.416 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:27:56.416 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:27:56.417 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:27:56.417 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:27:56.417 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:27:56.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:56.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:56.418 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:27:56.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:27:56.418 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:27:56.418 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:27:56.419 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:27:56.419 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:27:56.419 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:27:56.420 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:27:56.421 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:27:56.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:56.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:27:56.422 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:27:56.472 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:56.473 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:56.536 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:27:56.537 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:27:56.537 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:27:56.537 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:27:56.537 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:27:56.538 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:27:56.538 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:56.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:56.538 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:27:56.539 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:56.539 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:27:56.539 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:56.539 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:56.539 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:27:56.540 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:56.540 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:27:56.540 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:56.540 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:27:56.541 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:56.541 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:56.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:56.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:56.542 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:27:56.542 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:56.542 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:56.542 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:27:56.542 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:56.543 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:56.543 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:56.543 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:27:56.543 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:56.544 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:56.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:56.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:56.544 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:27:56.544 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:56.545 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:56.545 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927665.068925_56.1", 2021-09-06T11:27:56.545 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:56.545 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:56.546 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:56.546 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:45.068925Z", 2021-09-06T11:27:56.546 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:56.546 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:56.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:56.547 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:27:56.547 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:27:56.547 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:27:56.547 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:27:56.548 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:27:56.548 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:27:56.548 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:27:56.548 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:27:56.549 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:27:56.549 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:27:56.549 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:56.549 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:56.549 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:27:56.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:27:56.550 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:27:56.550 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:27:56.550 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:27:56.551 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:27:56.551 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:27:56.552 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:27:56.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:27:56.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:27:56.554 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:27:56.603 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:56.603 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:56.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:56.624+0000 7f74e2f15380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:27:56.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:56.624+0000 7f74e2f15380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:27:56.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:56.624+0000 7f74e2f15380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:27:56.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:56.624+0000 7f74e2f15380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:27:56.656 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:56.655+0000 7f74e2f15380 1 bucket sync caught up with source: 2021-09-06T11:27:56.657 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:27:56.657 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:27:56.657 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:56.655+0000 7f74e2f15380 0 bucket checkpoint complete 2021-09-06T11:27:56.671 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-10 zones={test-zone2, test-zone1} 2021-09-06T11:27:56.678 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:27:56.682 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:27:56.683 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:27:56.683 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:27:56.683 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:27:56.684 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:27:56.687 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:27:56.687 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:27:56.688 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:27:56.688 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:27:56.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:27:56.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:27:56.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:27:56.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:27:56.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:27:57.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:27:57.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:27:57.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:27:57.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:27:57.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:27:57.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:27:57.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:27:57.694 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-10 zones={test-zone2, test-zone1} 2021-09-06T11:27:57.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:27:57.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:27:57.696 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:27:57.750 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:57.750 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:57.774 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:27:57.774+0000 7f2bb60fe380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:27:57.775 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:27:57.774+0000 7f2bb60fe380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:27:57.775 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:27:57.774+0000 7f2bb60fe380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:27:57.775 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:27:57.774+0000 7f2bb60fe380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:27:57.805 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:27:57.805+0000 7f2bb60fe380 1 bucket sync caught up with source: 2021-09-06T11:27:57.806 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:27:57.806 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:27:57.806 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:27:57.805+0000 7f2bb60fe380 0 bucket checkpoint complete 2021-09-06T11:27:57.822 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-10 zones={test-zone3, test-zone1} 2021-09-06T11:27:57.828 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:27:57.831 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:27:57.832 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:27:57.832 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:27:57.833 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:27:57.833 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:27:57.837 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:27:57.837 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:27:57.838 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:27:57.838 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:27:57.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:27:57.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:27:57.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:27:58.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:27:58.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:27:58.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:27:58.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:27:58.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:27:58.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:27:58.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:27:58.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:27:58.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:27:58.834 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-10 zones={test-zone3, test-zone1} 2021-09-06T11:27:58.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:27:58.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:27:58.835 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:27:58.891 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:27:58.891 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:27:58.916 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:58.915+0000 7f721e074380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:27:58.916 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:58.915+0000 7f721e074380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:27:58.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:58.915+0000 7f721e074380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:27:58.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:58.915+0000 7f721e074380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:27:58.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:27:58.947+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:27:58.948 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:27:58.948 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:03.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:03.949+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:03.950 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:03.951 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:08.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:08.951+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:08.952 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:08.952 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:13.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:13.953+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:13.954 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:13.954 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:18.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:18.955+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:18.955 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:18.956 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:23.957 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:23.957+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:23.957 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:23.958 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:28.958 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:28.958+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:28.959 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:28.959 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:33.960 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:33.960+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:33.961 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:33.961 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:38.962 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:38.962+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:38.962 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:38.962 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:43.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:43.963+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:43.964 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:43.964 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:48.966 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:48.965+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:48.966 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:48.966 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:53.967 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:53.966+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:53.968 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:53.968 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:28:58.969 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:28:58.969+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:28:58.970 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:28:58.970 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:03.971 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:03.971+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:03.971 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:03.972 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:08.973 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:08.973+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:08.973 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:08.974 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:13.975 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:13.975+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:13.975 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:13.976 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:18.976 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:18.977+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:18.977 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:18.977 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:23.978 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:23.978+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:23.979 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:23.979 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:28.726 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:29:28.725+0000 7fc4b2a0c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T11:29:28.980 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:28.980+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:28.981 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:28.981 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:33.982 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:33.981+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:33.982 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:33.983 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:38.984 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:38.983+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:38.984 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:38.985 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:43.986 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:43.985+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:43.986 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:43.986 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:48.988 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:48.988+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:48.988 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:48.988 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:53.990 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:53.989+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:53.990 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:53.991 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:29:58.991 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:29:58.991+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:29:58.991 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:29:58.992 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:03.993 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:03.993+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:03.993 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:03.993 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:08.995 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:08.995+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:08.995 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:08.995 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:13.997 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:13.997+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:13.997 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:13.997 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:18.998 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:18.998+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:18.999 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:18.999 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:24.000 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:23.999+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:24.001 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:24.001 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:29.002 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:29.001+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:29.003 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:29.003 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:34.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:34.004+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:34.004 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:34.005 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:39.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:39.006+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:39.006 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:39.006 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:44.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:44.008+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:44.008 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:44.008 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:49.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:49.009+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:49.010 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:49.010 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:54.011 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:54.011+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:54.011 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:54.012 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:30:59.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:30:59.013+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:30:59.013 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:30:59.014 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:04.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:04.014+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:31:04.015 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:31:04.016 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:09.017 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:09.016+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:31:09.017 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:31:09.018 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:14.019 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:14.018+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:31:14.019 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:31:14.020 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:19.021 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:19.021+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:31:19.021 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:31:19.021 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:24.022 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:24.023+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:31:24.023 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:31:24.023 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:29.024 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:29.024+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:31:29.025 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:31:29.025 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:34.026 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:34.026+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:31:34.026 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:31:34.027 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:39.028 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:39.028+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:31:39.029 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:31:39.029 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:44.030 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:44.030+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:31:44.030 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:31:44.030 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:49.032 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:49.032+0000 7f721e074380 1 waiting for incremental sync to catch up: 2021-09-06T11:31:49.032 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, , , ] 2021-09-06T11:31:49.032 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:54.034 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:54.033+0000 7f721e074380 1 bucket sync caught up with source: 2021-09-06T11:31:54.034 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:54.035 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.6.5, , 00000000001.7.5, , , , 00000000001.4.5, 00000000001.9.5, , ] 2021-09-06T11:31:54.035 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:54.033+0000 7f721e074380 0 bucket checkpoint complete 2021-09-06T11:31:54.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:31:54.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:31:54.055 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:31:54.111 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:31:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:31:54.137 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:54.137+0000 7efdab12c380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:31:54.138 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:54.137+0000 7efdab12c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:31:54.138 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:54.137+0000 7efdab12c380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:31:54.138 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:54.137+0000 7efdab12c380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:31:54.170 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:54.169+0000 7efdab12c380 1 bucket sync caught up with source: 2021-09-06T11:31:54.170 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.23.5, , 00000000001.75.5, , , , 00000000001.32.5, 00000000001.75.5, , ] 2021-09-06T11:31:54.170 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.23.5, , 00000000001.75.5, , , , 00000000001.32.5, 00000000001.75.5, , ] 2021-09-06T11:31:54.170 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:54.169+0000 7efdab12c380 0 bucket checkpoint complete 2021-09-06T11:31:54.185 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-11 zones={test-zone1, test-zone2} 2021-09-06T11:31:54.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:31:54.201 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:31:54.201 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:31:54.202 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:31:54.202 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:31:54.203 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:31:54.206 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:31:54.207 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:31:54.207 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:31:54.208 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:31:54.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:54.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:54.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:54.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:54.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:54.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:54.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:54.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:54.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:54.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:55.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:55.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:55.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-11 zones={test-zone1, test-zone2} 2021-09-06T11:31:55.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:31:55.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:31:55.208 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:31:55.263 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:31:55.263 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:31:55.288 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:55.288+0000 7f6662e35380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:31:55.289 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:55.288+0000 7f6662e35380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:31:55.289 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:55.288+0000 7f6662e35380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:31:55.289 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:55.288+0000 7f6662e35380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:31:55.321 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:55.321+0000 7f6662e35380 1 bucket sync caught up with source: 2021-09-06T11:31:55.321 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.23.5, , 00000000001.75.5, , , , 00000000001.32.5, 00000000001.75.5, , ] 2021-09-06T11:31:55.322 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.23.5, , 00000000001.75.5, , , , 00000000001.32.5, 00000000001.75.5, , ] 2021-09-06T11:31:55.322 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:55.321+0000 7f6662e35380 0 bucket checkpoint complete 2021-09-06T11:31:55.336 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-11 zones={test-zone3, test-zone2} 2021-09-06T11:31:55.343 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:31:55.347 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:31:55.347 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:31:55.348 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:31:55.348 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:31:55.349 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:31:55.352 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:31:55.353 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:31:55.353 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:31:55.353 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:31:55.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:55.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:55.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:55.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:55.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:55.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:55.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:55.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:56.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:56.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:56.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:56.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:56.375 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-11 zones={test-zone3, test-zone2} 2021-09-06T11:31:56.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:31:56.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:31:56.376 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:31:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:31:56.429 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:31:56.453 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:56.453+0000 7f3cedd15380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:31:56.454 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:56.453+0000 7f3cedd15380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:31:56.454 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:56.453+0000 7f3cedd15380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:31:56.454 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:56.453+0000 7f3cedd15380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:31:56.485 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:56.485+0000 7f3cedd15380 1 bucket sync caught up with source: 2021-09-06T11:31:56.485 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.58.5, , 00000000001.43.5, , , , 00000000001.77.5, 00000000001.34.5, , ] 2021-09-06T11:31:56.485 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.58.5, , 00000000001.43.5, , , , 00000000001.77.5, 00000000001.34.5, , ] 2021-09-06T11:31:56.485 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:31:56.485+0000 7f3cedd15380 0 bucket checkpoint complete 2021-09-06T11:31:56.499 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-12 zones={test-zone1, test-zone3} 2021-09-06T11:31:56.505 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:31:56.508 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:31:56.509 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:31:56.509 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:31:56.510 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:31:56.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:31:56.514 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:31:56.514 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:31:56.515 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:31:56.515 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:31:56.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:56.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:56.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:56.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:56.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:56.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:57.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:57.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:57.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:57.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:57.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:57.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:57.518 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-12 zones={test-zone1, test-zone3} 2021-09-06T11:31:57.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:31:57.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:31:57.520 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:31:57.576 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:31:57.577 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:31:57.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:57.601+0000 7fcaeb675380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:31:57.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:57.601+0000 7fcaeb675380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:31:57.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:57.601+0000 7fcaeb675380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:31:57.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:57.601+0000 7fcaeb675380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:31:57.643 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:57.643+0000 7fcaeb675380 1 bucket sync caught up with source: 2021-09-06T11:31:57.643 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.58.5, , 00000000001.43.5, , , , 00000000001.77.5, 00000000001.34.5, , ] 2021-09-06T11:31:57.644 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.58.5, , 00000000001.43.5, , , , 00000000001.77.5, 00000000001.34.5, , ] 2021-09-06T11:31:57.644 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:31:57.643+0000 7fcaeb675380 0 bucket checkpoint complete 2021-09-06T11:31:57.659 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-12 zones={test-zone2, test-zone3} 2021-09-06T11:31:57.666 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:31:57.669 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:31:57.670 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:31:57.670 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:31:57.671 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:31:57.671 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:31:57.719 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T11:31:57.719 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T11:31:57.720 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T11:31:57.720 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:31:57.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:57.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:57.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T11:31:57.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:58.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:58.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T11:31:58.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:58.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:58.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T11:31:58.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:58.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:58.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:31:58.730 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-12 zones={test-zone2, test-zone3} 2021-09-06T11:31:58.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2021-09-06T11:31:58.732 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-13 2021-09-06T11:31:58.752 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-14 2021-09-06T11:31:58.786 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-15 2021-09-06T11:31:58.929 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:31:58.929+0000 7fc4c6a34700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:31:58.930 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2021-09-06T11:31:58.929+0000 7f1426e75700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:31:58.930 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2021-09-06T11:31:58.929+0000 7f3e4d2ac700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:31:58.932 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:31:58.931+0000 7fc4bf225700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:31:58.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:31:58.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:31:58.964 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:31:58.993 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2021-09-06T11:31:58.992+0000 7f3e522b6700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:31:59.008 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:31:59.008 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:31:59.055 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:31:59.072 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:31:59.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:31:59.072 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:31:59.073 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:31:59.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:31:59.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:31:59.073 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:31:59.074 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:31:59.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:31:59.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:31:59.074 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927918.806855_71.1", 2021-09-06T11:31:59.075 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:31:58.806855Z" 2021-09-06T11:31:59.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:31:59.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:31:59.075 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:31:59.075 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:31:59.076 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:31:59.076 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:31:59.077 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:31:59.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:31:59.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:31:59.079 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:31:59.124 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:31:59.124 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:31:59.185 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:31:59.185 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:31:59.185 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:31:59.186 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:31:59.186 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:31:59.186 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:31:59.187 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:31:59.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:31:59.187 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:31:59.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:31:59.187 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:31:59.188 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:31:59.188 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:31:59.188 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:31:59.188 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:31:59.189 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:31:59.189 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:31:59.189 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:31:59.189 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:31:59.189 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:31:59.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:31:59.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:31:59.190 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:31:59.190 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:31:59.191 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:31:59.191 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:31:59.191 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:31:59.191 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:31:59.192 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:31:59.192 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:31:59.192 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:31:59.192 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:31:59.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:31:59.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:31:59.193 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:31:59.193 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:31:59.193 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:31:59.194 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927665.068925_56.1", 2021-09-06T11:31:59.194 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:31:59.194 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:31:59.194 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:31:59.194 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:45.068925Z", 2021-09-06T11:31:59.195 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:31:59.195 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:31:59.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:31:59.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:31:59.196 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:31:59.196 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:31:59.196 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:31:59.196 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:31:59.196 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:31:59.197 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:31:59.197 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:31:59.197 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:31:59.197 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:31:59.198 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:31:59.198 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:31:59.198 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:31:59.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:31:59.198 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:31:59.199 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:31:59.199 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:31:59.199 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:31:59.199 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:31:59.200 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630927918.806855_71.1 target=1_1630927665.068925_56.1 2021-09-06T11:32:04.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:04.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:04.204 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:32:04.269 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:04.270 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:04.535 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:32:04.536 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:32:04.536 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:32:04.536 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:32:04.536 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:32:04.537 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:32:04.537 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:04.537 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:04.537 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:32:04.538 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:04.538 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:32:04.538 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:04.538 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:04.538 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:32:04.539 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:04.539 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:32:04.539 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:04.539 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:32:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:32:04.541 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:04.541 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:04.541 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:32:04.541 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:04.542 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:04.542 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:04.542 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:32:04.542 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:04.542 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:04.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:04.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:04.543 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:32:04.543 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:04.543 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:04.544 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927918.741184_60.1", 2021-09-06T11:32:04.544 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:04.544 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:04.544 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:04.545 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:31:58.741184Z", 2021-09-06T11:32:04.545 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:04.545 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:04.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:04.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:04.546 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:32:04.546 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:04.546 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:04.546 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:32:04.547 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:04.547 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:04.547 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:04.547 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:32:04.548 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:04.548 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:04.548 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:04.548 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:32:04.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:04.549 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:32:04.549 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:32:04.549 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:32:04.549 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:04.550 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:32:04.551 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630927918.806855_71.1 target=1_1630927918.741184_60.1 2021-09-06T11:32:09.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:09.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:09.553 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:32:09.602 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:09.602 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:09.673 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:32:09.673 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:32:09.674 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:32:09.674 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:32:09.674 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:32:09.674 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:32:09.675 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:09.675 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:09.675 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:32:09.675 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:09.676 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:32:09.676 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:09.676 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:09.676 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:32:09.677 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:09.677 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:32:09.677 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:09.677 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:32:09.677 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:09.678 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.678 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:09.678 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:09.678 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:32:09.678 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:09.679 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:09.679 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:32:09.679 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:09.679 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:09.680 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:09.680 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:32:09.680 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:09.680 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.680 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:09.681 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:09.681 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:32:09.681 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:09.681 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:09.681 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927918.806855_71.1", 2021-09-06T11:32:09.682 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:09.682 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:09.682 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:09.682 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:31:58.806855Z", 2021-09-06T11:32:09.683 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:09.683 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.683 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:09.683 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:09.684 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:32:09.684 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:09.684 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:09.684 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:32:09.684 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:09.685 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:09.685 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:09.685 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:32:09.685 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:09.686 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.686 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.686 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:32:09.686 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:09.687 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:32:09.687 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:32:09.687 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:32:09.687 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.688 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:32:09.689 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:32:09.690 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:32:09.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:09.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:09.691 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:32:09.738 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:09.738 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:09.802 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:32:09.803 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:32:09.803 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:32:09.803 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:32:09.803 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:32:09.804 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:32:09.804 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:32:09.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:09.804 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:32:09.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:09.805 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:32:09.805 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:32:09.805 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:32:09.805 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:32:09.806 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:32:09.806 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:32:09.806 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:32:09.806 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:32:09.807 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:32:09.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:09.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:09.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:09.808 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:32:09.808 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:32:09.808 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:32:09.808 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:32:09.808 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:32:09.809 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:32:09.809 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:32:09.809 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:32:09.809 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:32:09.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:09.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:09.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:09.810 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:32:09.810 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:32:09.811 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:32:09.811 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927918.806855_71.1", 2021-09-06T11:32:09.811 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:32:09.811 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:32:09.812 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:32:09.812 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:31:58.806855Z", 2021-09-06T11:32:09.812 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:32:09.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:09.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:09.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:09.813 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:32:09.814 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:32:09.814 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:32:09.814 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:32:09.814 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:32:09.814 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:32:09.815 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:32:09.815 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:32:09.815 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:32:09.815 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:09.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:09.816 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:32:09.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:09.816 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:32:09.817 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:32:09.817 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:32:09.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:09.817 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:32:09.818 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:32:09.819 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:32:09.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:09.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:09.821 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:32:09.866 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:09.866 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:09.937 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:32:09.938 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:32:09.938 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:32:09.938 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:32:09.938 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:32:09.939 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:32:09.939 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:09.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:09.939 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:32:09.940 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:09.940 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:32:09.940 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:09.940 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:09.941 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:32:09.941 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:09.941 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:32:09.941 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:09.942 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:32:09.942 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:09.942 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:09.943 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:09.943 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:32:09.943 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:09.943 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:09.944 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:32:09.944 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:09.944 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:09.944 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:09.945 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:32:09.945 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:09.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:09.946 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:09.946 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:32:09.946 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:09.946 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:09.947 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927918.806855_71.1", 2021-09-06T11:32:09.947 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:09.947 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:09.947 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:09.948 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:31:58.806855Z", 2021-09-06T11:32:09.948 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:09.948 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.948 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:09.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:09.949 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:32:09.949 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:09.949 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:09.949 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:32:09.950 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:09.950 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:09.950 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:09.950 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:32:09.951 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:09.951 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.951 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.951 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:32:09.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:09.952 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:32:09.952 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:32:09.952 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:32:09.953 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:09.953 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:32:09.954 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:32:09.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:09.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:09.956 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:10.004 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:10.005 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:10.027 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:10.027+0000 7fc14c662380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:10.027 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:10.027+0000 7fc14c662380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:10.027 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:10.027+0000 7fc14c662380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:32:10.028 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:10.027+0000 7fc14c662380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:10.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:10.065+0000 7fc14c662380 1 bucket sync caught up with source: 2021-09-06T11:32:10.065 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2021-09-06T11:32:10.065 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2021-09-06T11:32:10.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:10.065+0000 7fc14c662380 0 bucket checkpoint complete 2021-09-06T11:32:10.079 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-13 zones={test-zone2, test-zone1} 2021-09-06T11:32:10.085 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:10.131 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:10.132 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:10.135 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:10.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:10.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:10.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:10.442 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-13 zones={test-zone2, test-zone1} 2021-09-06T11:32:10.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:10.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:10.444 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:10.498 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:10.498 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:10.523 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:10.523+0000 7f7d4898e380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:10.523+0000 7f7d4898e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:10.523+0000 7f7d4898e380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:32:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:10.523+0000 7f7d4898e380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:10.555 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:10.554+0000 7f7d4898e380 1 bucket sync caught up with source: 2021-09-06T11:32:10.555 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2021-09-06T11:32:10.555 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2021-09-06T11:32:10.556 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:10.554+0000 7f7d4898e380 0 bucket checkpoint complete 2021-09-06T11:32:10.574 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-13 zones={test-zone3, test-zone1} 2021-09-06T11:32:10.581 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:10.631 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:10.631 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:10.635 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:10.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:10.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:10.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:10.930 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-13 zones={test-zone3, test-zone1} 2021-09-06T11:32:10.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:10.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:10.931 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:10.984 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:10.984 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:11.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:11.009+0000 7f24bbfea380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:11.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:11.009+0000 7f24bbfea380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:11.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:11.009+0000 7f24bbfea380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:32:11.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:11.009+0000 7f24bbfea380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:11.041 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:11.042+0000 7f24bbfea380 1 bucket sync caught up with source: 2021-09-06T11:32:11.042 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2021-09-06T11:32:11.042 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2021-09-06T11:32:11.042 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:11.042+0000 7f24bbfea380 0 bucket checkpoint complete 2021-09-06T11:32:11.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:11.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:11.057 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:11.107 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:11.107 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:11.129 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.129+0000 7f404a4fb380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:11.129 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.129+0000 7f404a4fb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:11.129 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.129+0000 7f404a4fb380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:32:11.129 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.129+0000 7f404a4fb380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:11.159 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.159+0000 7f404a4fb380 1 bucket sync caught up with source: 2021-09-06T11:32:11.159 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.76.5, , , , , , , , , ] 2021-09-06T11:32:11.160 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.76.5, , , , , , , , , ] 2021-09-06T11:32:11.160 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.159+0000 7f404a4fb380 0 bucket checkpoint complete 2021-09-06T11:32:11.174 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-14 zones={test-zone1, test-zone2} 2021-09-06T11:32:11.180 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:11.227 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:11.228 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:11.231 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:11.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:11.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:11.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:11.534 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-14 zones={test-zone1, test-zone2} 2021-09-06T11:32:11.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:11.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:11.536 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:11.587 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:11.588 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:11.611 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.611+0000 7f2b8a149380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:11.611 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.611+0000 7f2b8a149380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:11.611 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.611+0000 7f2b8a149380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:32:11.612 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.611+0000 7f2b8a149380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:11.643 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.643+0000 7f2b8a149380 1 bucket sync caught up with source: 2021-09-06T11:32:11.643 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.76.5, , , , , , , , , ] 2021-09-06T11:32:11.643 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.76.5, , , , , , , , , ] 2021-09-06T11:32:11.643 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:11.643+0000 7f2b8a149380 0 bucket checkpoint complete 2021-09-06T11:32:11.657 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-14 zones={test-zone3, test-zone2} 2021-09-06T11:32:11.663 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:11.711 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:11.711 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:11.715 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:11.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:11.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:11.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:12.046 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-14 zones={test-zone3, test-zone2} 2021-09-06T11:32:12.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:12.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:12.048 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:12.096 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:12.119 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:12.119+0000 7f0342765380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:12.119 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:12.119+0000 7f0342765380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:12.120 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:12.119+0000 7f0342765380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:32:12.120 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:12.119+0000 7f0342765380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:12.149 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:12.149+0000 7f0342765380 1 bucket sync caught up with source: 2021-09-06T11:32:12.149 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.192.5, , , , , , , , , ] 2021-09-06T11:32:12.150 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.192.5, , , , , , , , , ] 2021-09-06T11:32:12.150 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:12.149+0000 7f0342765380 0 bucket checkpoint complete 2021-09-06T11:32:12.164 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-15 zones={test-zone1, test-zone3} 2021-09-06T11:32:12.170 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:12.215 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:12.216 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:12.219 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:12.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:12.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:12.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:12.518 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-15 zones={test-zone1, test-zone3} 2021-09-06T11:32:12.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:12.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:12.520 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:12.575 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:12.576 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:12.603 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:12.603+0000 7fe8ea665380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:12.603 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:12.603+0000 7fe8ea665380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:12.603 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:12.603+0000 7fe8ea665380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:32:12.603 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:12.603+0000 7fe8ea665380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:12.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:12.641+0000 7fe8ea665380 1 bucket sync caught up with source: 2021-09-06T11:32:12.641 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.192.5, , , , , , , , , ] 2021-09-06T11:32:12.642 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000001.192.5, , , , , , , , , ] 2021-09-06T11:32:12.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:12.641+0000 7fe8ea665380 0 bucket checkpoint complete 2021-09-06T11:32:12.657 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-15 zones={test-zone2, test-zone3} 2021-09-06T11:32:12.663 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:12.707 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:12.708 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:12.711 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T11:32:12.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:12.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:12.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T11:32:13.018 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-15 zones={test-zone2, test-zone3} 2021-09-06T11:32:13.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:13.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:13.029 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:13.085 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:13.085 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:13.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:13.111+0000 7fded386c380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:13.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:13.111+0000 7fded386c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:13.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:13.111+0000 7fded386c380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:32:13.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:13.111+0000 7fded386c380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:13.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:13.151+0000 7fded386c380 1 waiting for incremental sync to catch up: 2021-09-06T11:32:13.151 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2021-09-06T11:32:13.151 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2021-09-06T11:32:18.153 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:18.153+0000 7fded386c380 1 bucket sync caught up with source: 2021-09-06T11:32:18.154 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000002.11.5, , , , , , , , , ] 2021-09-06T11:32:18.154 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2021-09-06T11:32:18.154 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:18.153+0000 7fded386c380 0 bucket checkpoint complete 2021-09-06T11:32:18.172 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-13 zones={test-zone2, test-zone1} 2021-09-06T11:32:18.179 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:18.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:18.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-13 zones={test-zone2, test-zone1} 2021-09-06T11:32:18.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:18.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:18.276 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:18.331 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:18.331 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:18.355 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.355+0000 7fb4dbef1380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:18.356 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.355+0000 7fb4dbef1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:18.356 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.355+0000 7fb4dbef1380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:32:18.356 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.355+0000 7fb4dbef1380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:18.387 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.387+0000 7fb4dbef1380 1 bucket sync caught up with source: 2021-09-06T11:32:18.387 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000002.11.5, , , , , , , , , ] 2021-09-06T11:32:18.388 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2021-09-06T11:32:18.388 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.387+0000 7fb4dbef1380 0 bucket checkpoint complete 2021-09-06T11:32:18.403 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-13 zones={test-zone3, test-zone1} 2021-09-06T11:32:18.409 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:18.455 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:18.505 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-13 zones={test-zone3, test-zone1} 2021-09-06T11:32:18.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:18.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:18.507 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:18.564 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:18.564 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:18.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:18.589+0000 7f0b5176b380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:18.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:18.589+0000 7f0b5176b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:18.590 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:18.589+0000 7f0b5176b380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:32:18.590 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:18.589+0000 7f0b5176b380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:18.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:18.624+0000 7f0b5176b380 1 bucket sync caught up with source: 2021-09-06T11:32:18.624 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000002.11.5, , , , , , , , , ] 2021-09-06T11:32:18.624 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2021-09-06T11:32:18.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:18.624+0000 7f0b5176b380 0 bucket checkpoint complete 2021-09-06T11:32:18.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:18.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:18.653 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:18.704 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:18.704 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:18.727 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.727+0000 7fe92ae33380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:18.727 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.727+0000 7fe92ae33380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:18.728 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.727+0000 7fe92ae33380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:32:18.728 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.727+0000 7fe92ae33380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:18.758 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:18.758+0000 7fe92ae33380 1 waiting for incremental sync to catch up: 2021-09-06T11:32:18.758 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.76.5, , , , , , , , , ] 2021-09-06T11:32:18.758 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.338.5, , , , , , , , , ] 2021-09-06T11:32:23.760 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:23.760+0000 7fe92ae33380 1 bucket sync caught up with source: 2021-09-06T11:32:23.760 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000002.338.5, , , , , , , , , ] 2021-09-06T11:32:23.761 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.338.5, , , , , , , , , ] 2021-09-06T11:32:23.761 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:23.760+0000 7fe92ae33380 0 bucket checkpoint complete 2021-09-06T11:32:23.778 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-14 zones={test-zone1, test-zone2} 2021-09-06T11:32:23.785 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:23.831 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:23.882 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-14 zones={test-zone1, test-zone2} 2021-09-06T11:32:23.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:23.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:23.883 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:23.939 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:23.939 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:23.964 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:23.964+0000 7fc91fd34380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:23.964 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:23.964+0000 7fc91fd34380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:23.965 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:23.964+0000 7fc91fd34380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:32:23.965 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:23.964+0000 7fc91fd34380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:23.996 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:23.996+0000 7fc91fd34380 1 bucket sync caught up with source: 2021-09-06T11:32:23.996 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000002.338.5, , , , , , , , , ] 2021-09-06T11:32:23.996 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.338.5, , , , , , , , , ] 2021-09-06T11:32:23.997 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:23.996+0000 7fc91fd34380 0 bucket checkpoint complete 2021-09-06T11:32:24.012 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-14 zones={test-zone3, test-zone2} 2021-09-06T11:32:24.018 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:24.063 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:24.114 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-14 zones={test-zone3, test-zone2} 2021-09-06T11:32:24.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:24.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:24.125 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:24.196 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:24.196+0000 7f5dc43b1380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:24.196 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:24.196+0000 7f5dc43b1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:24.196 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:24.196+0000 7f5dc43b1380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:32:24.197 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:24.196+0000 7f5dc43b1380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:24.227 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:24.227+0000 7f5dc43b1380 1 waiting for incremental sync to catch up: 2021-09-06T11:32:24.227 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.192.5, , , , , , , , , ] 2021-09-06T11:32:24.228 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.204.5, , , , , , , , , ] 2021-09-06T11:32:29.229 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:29.229+0000 7f5dc43b1380 1 bucket sync caught up with source: 2021-09-06T11:32:29.229 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000002.204.5, , , , , , , , , ] 2021-09-06T11:32:29.229 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.204.5, , , , , , , , , ] 2021-09-06T11:32:29.230 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:29.229+0000 7f5dc43b1380 0 bucket checkpoint complete 2021-09-06T11:32:29.247 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-15 zones={test-zone1, test-zone3} 2021-09-06T11:32:29.254 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:29.299 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:29.350 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-15 zones={test-zone1, test-zone3} 2021-09-06T11:32:29.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:29.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:29.352 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:29.406 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:29.407 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:29.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:29.432+0000 7f2386ad3380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:29.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:29.432+0000 7f2386ad3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:29.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:29.432+0000 7f2386ad3380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:32:29.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:29.432+0000 7f2386ad3380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:29.470 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:29.471+0000 7f2386ad3380 1 bucket sync caught up with source: 2021-09-06T11:32:29.470 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000002.204.5, , , , , , , , , ] 2021-09-06T11:32:29.471 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000002.204.5, , , , , , , , , ] 2021-09-06T11:32:29.471 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:29.471+0000 7f2386ad3380 0 bucket checkpoint complete 2021-09-06T11:32:29.491 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-15 zones={test-zone2, test-zone3} 2021-09-06T11:32:29.497 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:29.543 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:29.594 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-15 zones={test-zone2, test-zone3} 2021-09-06T11:32:29.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2021-09-06T11:32:29.596 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-16 2021-09-06T11:32:29.616 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-17 2021-09-06T11:32:29.651 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-18 2021-09-06T11:32:29.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:29.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:29.712 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:32:29.759 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:29.760 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:29.805 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:32:29.820 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:32:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:32:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:32:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:32:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:32:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927949.708878_89.1", 2021-09-06T11:32:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:32:29.708878Z" 2021-09-06T11:32:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:32:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:32:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:29.824 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:32:29.825 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:32:29.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:29.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:29.827 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:32:29.870 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:29.870 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:29.931 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:32:29.931 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:32:29.931 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:32:29.932 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:32:29.932 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:32:29.932 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:32:29.932 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:29.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:29.933 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:32:29.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:29.933 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:32:29.933 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:29.934 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:29.934 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:32:29.934 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:29.934 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:32:29.935 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:29.935 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:32:29.935 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:29.935 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:29.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:29.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:29.936 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:32:29.936 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:29.936 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:29.937 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:32:29.937 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:29.937 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:29.937 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:29.938 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:32:29.938 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:29.938 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:29.938 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:29.938 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:29.939 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:32:29.939 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:29.939 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:29.939 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927949.605652_72.1", 2021-09-06T11:32:29.940 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:29.940 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:29.940 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:29.940 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:32:29.605652Z", 2021-09-06T11:32:29.941 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:29.941 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:29.941 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:29.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:29.941 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:32:29.942 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:29.942 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:29.942 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:32:29.942 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:29.943 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:29.943 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:29.943 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:32:29.943 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:29.943 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:29.944 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:29.944 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:32:29.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:29.944 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:32:29.945 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:32:29.945 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:32:29.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:29.945 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:32:29.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630927949.708878_89.1 target=1_1630927949.605652_72.1 2021-09-06T11:32:34.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:34.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:34.949 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:32:34.997 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:34.998 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:35.064 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:32:35.064 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:32:35.064 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:32:35.065 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:32:35.065 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:32:35.065 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:32:35.065 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:35.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:35.066 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:32:35.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:35.066 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:32:35.066 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:35.067 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:35.067 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:32:35.067 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:35.067 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:32:35.067 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:35.068 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:32:35.068 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:35.068 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:35.068 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:35.069 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:32:35.069 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:35.069 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:35.069 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:32:35.069 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:35.070 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:35.070 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:35.070 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:32:35.070 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:35.071 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:35.071 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:35.071 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:32:35.071 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:35.072 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:35.072 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927949.708878_89.1", 2021-09-06T11:32:35.072 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:35.072 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:35.072 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:35.073 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:32:29.708878Z", 2021-09-06T11:32:35.073 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:35.073 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:35.073 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:35.074 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:32:35.074 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:35.074 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:35.074 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:32:35.075 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:35.075 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:35.075 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:35.075 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:32:35.075 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:35.076 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.076 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.076 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:32:35.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:35.076 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:32:35.077 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:32:35.077 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:32:35.077 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.077 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:32:35.078 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:32:35.079 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:32:35.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:35.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:35.080 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:32:35.128 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:35.129 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:35.192 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:32:35.192 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:32:35.193 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:32:35.193 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:32:35.193 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:32:35.193 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:32:35.194 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:32:35.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:35.194 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:32:35.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:35.194 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:32:35.195 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:32:35.195 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:32:35.195 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:32:35.195 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:32:35.195 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:32:35.196 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:32:35.196 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:32:35.196 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:32:35.196 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:35.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:35.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:35.197 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:32:35.197 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:32:35.197 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:32:35.197 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:32:35.198 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:32:35.198 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:32:35.198 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:32:35.198 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:32:35.199 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:32:35.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:35.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:35.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:35.199 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:32:35.200 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:32:35.200 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:32:35.200 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927949.708878_89.1", 2021-09-06T11:32:35.200 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:32:35.200 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:32:35.201 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:32:35.201 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:32:29.708878Z", 2021-09-06T11:32:35.201 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:32:35.201 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:35.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:35.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:32:35.202 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:32:35.202 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:32:35.202 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:32:35.202 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:32:35.203 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:32:35.203 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:32:35.203 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:32:35.203 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:32:35.204 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:32:35.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:35.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:35.204 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:32:35.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:32:35.205 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:32:35.205 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:32:35.205 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:32:35.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:32:35.206 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:32:35.207 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:32:35.207 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:32:35.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:35.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:32:35.209 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:32:35.253 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:35.253 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:35.321 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:32:35.321 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:32:35.321 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:32:35.321 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:32:35.322 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:32:35.322 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:32:35.322 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:35.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:35.322 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:32:35.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:35.323 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:32:35.323 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:35.323 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:35.324 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:32:35.324 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:35.324 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:32:35.324 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:35.324 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:32:35.325 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:35.325 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:35.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:35.326 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:32:35.326 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:35.326 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:35.326 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:32:35.326 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:35.327 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:35.327 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:35.327 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:32:35.327 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:35.327 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:35.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:35.328 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:32:35.328 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:35.328 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:35.329 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927949.708878_89.1", 2021-09-06T11:32:35.329 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:35.329 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:35.329 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:35.330 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:32:29.708878Z", 2021-09-06T11:32:35.330 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:35.330 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:35.330 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:32:35.331 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:32:35.331 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:32:35.331 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:32:35.331 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:32:35.332 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:32:35.332 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:32:35.332 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:32:35.332 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:32:35.332 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:32:35.333 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.333 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.333 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:32:35.333 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:32:35.333 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:32:35.334 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:32:35.334 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:32:35.334 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:32:35.334 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:32:35.335 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:32:35.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:35.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:35.384 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:35.432 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:35.432 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:35.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:35.455+0000 7ff8fa809380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:35.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:35.455+0000 7ff8fa809380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:35.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:35.455+0000 7ff8fa809380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:32:35.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:35.455+0000 7ff8fa809380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:35.486 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:35.485+0000 7ff8fa809380 1 waiting for incremental sync to catch up: 2021-09-06T11:32:35.486 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , , , ] 2021-09-06T11:32:35.486 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2021-09-06T11:32:40.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:40.487+0000 7ff8fa809380 1 bucket sync caught up with source: 2021-09-06T11:32:40.489 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , 00000000005.14.9, , , , , ] 2021-09-06T11:32:40.489 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2021-09-06T11:32:40.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:40.487+0000 7ff8fa809380 0 bucket checkpoint complete 2021-09-06T11:32:40.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:40.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:40.509 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:40.567 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:40.567 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:40.593 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.594+0000 7f2ce775e380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:40.593 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.594+0000 7f2ce775e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:40.594 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.594+0000 7f2ce775e380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:32:40.594 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.594+0000 7f2ce775e380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.629+0000 7f2ce775e380 1 bucket sync caught up with source: 2021-09-06T11:32:40.628 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000005.14.9, , , , , ] 2021-09-06T11:32:40.628 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2021-09-06T11:32:40.628 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.629+0000 7f2ce775e380 0 bucket checkpoint complete 2021-09-06T11:32:40.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:40.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:40.644 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:40.699 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:40.699 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:40.723 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:40.722+0000 7fb95cd06380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:40.723 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:40.722+0000 7fb95cd06380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:40.724 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:40.722+0000 7fb95cd06380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:32:40.724 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:40.722+0000 7fb95cd06380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:40.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:40.758+0000 7fb95cd06380 1 bucket sync caught up with source: 2021-09-06T11:32:40.759 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , 00000000005.14.9, , , , , ] 2021-09-06T11:32:40.759 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2021-09-06T11:32:40.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:40.758+0000 7fb95cd06380 0 bucket checkpoint complete 2021-09-06T11:32:40.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:40.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:40.820 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:40.869 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:40.870 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:40.894 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.895+0000 7ff70b6c9380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:40.895 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.895+0000 7ff70b6c9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:40.895 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.895+0000 7ff70b6c9380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:32:40.895 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.895+0000 7ff70b6c9380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:40.924 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:40.925+0000 7ff70b6c9380 1 waiting for incremental sync to catch up: 2021-09-06T11:32:40.924 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , , , ] 2021-09-06T11:32:40.925 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000002.176.10, , , , , ] 2021-09-06T11:32:45.926 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:45.926+0000 7ff70b6c9380 1 bucket sync caught up with source: 2021-09-06T11:32:45.926 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000006.184.9, , , , , ] 2021-09-06T11:32:45.927 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000002.176.10, , , , , ] 2021-09-06T11:32:45.927 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:45.926+0000 7ff70b6c9380 0 bucket checkpoint complete 2021-09-06T11:32:45.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:45.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:45.946 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:45.998 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:45.998 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:46.024 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.023+0000 7f0187cfb380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:46.024 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.023+0000 7f0187cfb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:46.024 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.023+0000 7f0187cfb380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:32:46.024 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.023+0000 7f0187cfb380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:46.056 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.056+0000 7f0187cfb380 1 bucket sync caught up with source: 2021-09-06T11:32:46.056 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000006.184.9, , , , , ] 2021-09-06T11:32:46.056 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000006.184.9, , , , , ] 2021-09-06T11:32:46.057 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.056+0000 7f0187cfb380 0 bucket checkpoint complete 2021-09-06T11:32:46.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:46.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:46.120 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:46.170 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:46.170 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:46.192 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.192+0000 7ff0062fc380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.192+0000 7ff0062fc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.192+0000 7ff0062fc380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:32:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.192+0000 7ff0062fc380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:46.224 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:46.224+0000 7ff0062fc380 1 waiting for incremental sync to catch up: 2021-09-06T11:32:46.225 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , , , ] 2021-09-06T11:32:46.225 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000002.180.10, , , , , ] 2021-09-06T11:32:51.227 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:51.226+0000 7ff0062fc380 1 bucket sync caught up with source: 2021-09-06T11:32:51.227 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000006.188.9, , , , , ] 2021-09-06T11:32:51.227 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000002.180.10, , , , , ] 2021-09-06T11:32:51.227 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:51.226+0000 7ff0062fc380 0 bucket checkpoint complete 2021-09-06T11:32:51.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:51.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:51.246 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:51.304 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:51.304 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:51.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:51.335+0000 7f74fd55a380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:51.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:51.335+0000 7f74fd55a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:51.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:51.335+0000 7f74fd55a380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:32:51.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:51.335+0000 7f74fd55a380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:51.369 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:51.369+0000 7f74fd55a380 1 bucket sync caught up with source: 2021-09-06T11:32:51.369 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , 00000000006.188.9, , , , , ] 2021-09-06T11:32:51.370 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000006.188.9, , , , , ] 2021-09-06T11:32:51.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:51.369+0000 7f74fd55a380 0 bucket checkpoint complete 2021-09-06T11:32:52.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:52.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:52.858 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:52.911 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:52.911 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:52.934 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:52.934+0000 7f9557f30380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:52.934 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:52.934+0000 7f9557f30380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:52.934 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:52.934+0000 7f9557f30380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:32:52.935 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:52.934+0000 7f9557f30380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:52.971 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:52.971+0000 7f9557f30380 1 bucket sync caught up with source: 2021-09-06T11:32:52.971 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000010.32.5, , 00000000003.19.9, , 00000000008.20.9, , , , , ] 2021-09-06T11:32:52.971 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000010.32.5, , 00000000003.19.9, , 00000000008.20.9, , , , , ] 2021-09-06T11:32:52.971 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:52.971+0000 7f9557f30380 0 bucket checkpoint complete 2021-09-06T11:32:52.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:52.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:52.990 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:53.042 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:53.042 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:53.064 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.064+0000 7f683aa95380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:53.064 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.064+0000 7f683aa95380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:53.064 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.064+0000 7f683aa95380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:32:53.065 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.064+0000 7f683aa95380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:53.094 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.094+0000 7f683aa95380 1 bucket sync caught up with source: 2021-09-06T11:32:53.094 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000010.32.5, , 00000000003.19.9, , 00000000008.20.9, , , , , ] 2021-09-06T11:32:53.095 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000010.32.5, , 00000000003.19.9, , 00000000008.20.9, , , , , ] 2021-09-06T11:32:53.095 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.094+0000 7f683aa95380 0 bucket checkpoint complete 2021-09-06T11:32:53.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:53.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:53.110 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:53.162 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:53.162 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:53.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.184+0000 7fcabe33d380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:53.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.184+0000 7fcabe33d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:53.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.184+0000 7fcabe33d380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:32:53.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.184+0000 7fcabe33d380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:53.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.226+0000 7fcabe33d380 1 bucket sync caught up with source: 2021-09-06T11:32:53.226 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000010.32.5, , 00000000003.19.9, , 00000000008.20.9, , , , , ] 2021-09-06T11:32:53.227 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000010.32.5, , 00000000003.19.9, , 00000000008.20.9, , , , , ] 2021-09-06T11:32:53.227 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.226+0000 7fcabe33d380 0 bucket checkpoint complete 2021-09-06T11:32:53.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:53.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:53.243 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:53.291 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:53.315 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.314+0000 7fc9760af380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:53.315 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.314+0000 7fc9760af380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:53.315 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.314+0000 7fc9760af380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:32:53.315 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.314+0000 7fc9760af380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:53.346 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.346+0000 7fc9760af380 1 bucket sync caught up with source: 2021-09-06T11:32:53.346 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.190.9, , 00000000010.309.5, , 00000000007.135.9, , , , , ] 2021-09-06T11:32:53.347 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.190.9, , 00000000010.309.5, , 00000000007.135.9, , , , , ] 2021-09-06T11:32:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.346+0000 7fc9760af380 0 bucket checkpoint complete 2021-09-06T11:32:53.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:53.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:53.362 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:53.410 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:53.410 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:53.432 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.432+0000 7fef03dca380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:53.433 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.432+0000 7fef03dca380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:53.433 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.432+0000 7fef03dca380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:32:53.433 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.432+0000 7fef03dca380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:53.463 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.463+0000 7fef03dca380 1 bucket sync caught up with source: 2021-09-06T11:32:53.463 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.190.9, , 00000000010.309.5, , 00000000007.135.9, , , , , ] 2021-09-06T11:32:53.463 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.190.9, , 00000000010.309.5, , 00000000007.135.9, , , , , ] 2021-09-06T11:32:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.463+0000 7fef03dca380 0 bucket checkpoint complete 2021-09-06T11:32:53.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:53.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:53.479 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:53.555 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.555+0000 7fee0cc68380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:53.555 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.555+0000 7fee0cc68380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:53.556 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.555+0000 7fee0cc68380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:32:53.556 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.555+0000 7fee0cc68380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:53.586 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.586+0000 7fee0cc68380 1 bucket sync caught up with source: 2021-09-06T11:32:53.586 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.103.9, , 00000000003.232.9, , 00000000014.126.5, , , , , ] 2021-09-06T11:32:53.587 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.103.9, , 00000000003.232.9, , 00000000014.126.5, , , , , ] 2021-09-06T11:32:53.587 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:53.586+0000 7fee0cc68380 0 bucket checkpoint complete 2021-09-06T11:32:53.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:53.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:53.602 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:53.650 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:53.651 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:53.673 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.673+0000 7fd7e78bc380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:53.673 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.673+0000 7fd7e78bc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:53.674 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.673+0000 7fd7e78bc380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:32:53.674 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.673+0000 7fd7e78bc380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:53.709 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.709+0000 7fd7e78bc380 1 bucket sync caught up with source: 2021-09-06T11:32:53.709 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000003.103.9, , 00000000003.232.9, , 00000000014.126.5, , , , , ] 2021-09-06T11:32:53.709 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000003.103.9, , 00000000003.232.9, , 00000000014.126.5, , , , , ] 2021-09-06T11:32:53.710 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:53.709+0000 7fd7e78bc380 0 bucket checkpoint complete 2021-09-06T11:32:53.725 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-16 zones={test-zone2, test-zone1} 2021-09-06T11:32:53.731 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:53.735 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:53.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:53.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:53.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:53.737 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:53.740 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:53.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:53.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:53.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:53.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:53.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:53.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:53.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:53.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:54.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:54.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:54.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:54.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:54.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:54.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:54.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:54.610 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-16 zones={test-zone2, test-zone1} 2021-09-06T11:32:54.610 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-16 zones={test-zone3, test-zone1} 2021-09-06T11:32:54.619 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:54.622 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:54.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:54.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:54.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:54.624 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:54.627 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:54.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:54.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:54.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:54.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:54.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:54.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:54.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:54.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:54.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:54.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:55.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:55.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:55.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:55.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:55.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:55.486 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-16 zones={test-zone3, test-zone1} 2021-09-06T11:32:55.486 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-16 zones={test-zone3, test-zone2} 2021-09-06T11:32:55.492 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:55.496 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:55.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:55.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:55.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:55.497 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:55.501 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:55.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:55.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:55.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:55.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:55.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:55.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:55.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:55.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:55.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:55.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:55.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:56.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:56.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:56.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:56.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:56.363 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-16 zones={test-zone3, test-zone2} 2021-09-06T11:32:56.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.364 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:56.419 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:56.419 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:56.444 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.444+0000 7fcdd26ce380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:56.444 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.444+0000 7fcdd26ce380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:56.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.444+0000 7fcdd26ce380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:32:56.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.444+0000 7fcdd26ce380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:56.477 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.477+0000 7fcdd26ce380 1 bucket sync caught up with source: 2021-09-06T11:32:56.477 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000010.33.5, , 00000000003.9.9, , 00000000007.41.9, , , , , ] 2021-09-06T11:32:56.477 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000010.33.5, , 00000000003.9.9, , 00000000007.41.9, , , , , ] 2021-09-06T11:32:56.478 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.477+0000 7fcdd26ce380 0 bucket checkpoint complete 2021-09-06T11:32:56.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.494 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:56.543 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:56.543 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:56.568 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.568+0000 7f17b5217380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:56.569 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.568+0000 7f17b5217380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:56.569 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.568+0000 7f17b5217380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:32:56.569 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.568+0000 7f17b5217380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:56.599 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.599+0000 7f17b5217380 1 bucket sync caught up with source: 2021-09-06T11:32:56.599 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000010.33.5, , 00000000003.9.9, , 00000000007.41.9, , , , , ] 2021-09-06T11:32:56.599 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000010.33.5, , 00000000003.9.9, , 00000000007.41.9, , , , , ] 2021-09-06T11:32:56.599 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.599+0000 7f17b5217380 0 bucket checkpoint complete 2021-09-06T11:32:56.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.617 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:56.665 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:56.666 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:56.688 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.689+0000 7f6af7f58380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:56.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.689+0000 7f6af7f58380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:56.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.689+0000 7f6af7f58380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:32:56.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.689+0000 7f6af7f58380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:56.719 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.720+0000 7f6af7f58380 1 bucket sync caught up with source: 2021-09-06T11:32:56.719 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000010.33.5, , 00000000003.9.9, , 00000000007.41.9, , , , , ] 2021-09-06T11:32:56.720 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000010.33.5, , 00000000003.9.9, , 00000000007.41.9, , , , , ] 2021-09-06T11:32:56.720 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:56.720+0000 7f6af7f58380 0 bucket checkpoint complete 2021-09-06T11:32:56.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.735 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.806+0000 7fbc60a65380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.806+0000 7fbc60a65380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.806+0000 7fbc60a65380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:32:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.806+0000 7fbc60a65380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:56.836 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.836+0000 7fbc60a65380 1 bucket sync caught up with source: 2021-09-06T11:32:56.836 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.364.9, , 00000000010.120.5, , 00000000009.196.9, , , , , ] 2021-09-06T11:32:56.837 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.364.9, , 00000000010.120.5, , 00000000009.196.9, , , , , ] 2021-09-06T11:32:56.837 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.836+0000 7fbc60a65380 0 bucket checkpoint complete 2021-09-06T11:32:56.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.852 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:56.900 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:56.900 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.923+0000 7fb8fa240380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.923+0000 7fb8fa240380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:56.923 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.923+0000 7fb8fa240380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:32:56.924 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.923+0000 7fb8fa240380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:56.953 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.953+0000 7fb8fa240380 1 bucket sync caught up with source: 2021-09-06T11:32:56.954 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.364.9, , 00000000010.120.5, , 00000000009.196.9, , , , , ] 2021-09-06T11:32:56.954 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.364.9, , 00000000010.120.5, , 00000000009.196.9, , , , , ] 2021-09-06T11:32:56.954 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:56.953+0000 7fb8fa240380 0 bucket checkpoint complete 2021-09-06T11:32:56.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:56.969 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:57.018 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:57.018 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:57.042 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:57.042+0000 7fad0da69380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:57.042 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:57.042+0000 7fad0da69380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:57.043 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:57.042+0000 7fad0da69380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:32:57.043 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:57.042+0000 7fad0da69380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:57.072 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:57.072+0000 7fad0da69380 1 bucket sync caught up with source: 2021-09-06T11:32:57.073 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.151.9, , 00000000003.78.9, , 00000000014.123.5, , , , , ] 2021-09-06T11:32:57.073 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.151.9, , 00000000003.78.9, , 00000000014.123.5, , , , , ] 2021-09-06T11:32:57.073 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:32:57.072+0000 7fad0da69380 0 bucket checkpoint complete 2021-09-06T11:32:57.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:57.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:57.088 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:32:57.139 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:32:57.140 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:32:57.163 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:57.163+0000 7f44cff54380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:32:57.163 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:57.163+0000 7f44cff54380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:32:57.163 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:57.163+0000 7f44cff54380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:32:57.163 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:57.163+0000 7f44cff54380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:32:57.194 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:57.194+0000 7f44cff54380 1 bucket sync caught up with source: 2021-09-06T11:32:57.194 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000003.151.9, , 00000000003.78.9, , 00000000014.123.5, , , , , ] 2021-09-06T11:32:57.194 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000003.151.9, , 00000000003.78.9, , 00000000014.123.5, , , , , ] 2021-09-06T11:32:57.195 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:32:57.194+0000 7f44cff54380 0 bucket checkpoint complete 2021-09-06T11:32:57.211 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-17 zones={test-zone2, test-zone1} 2021-09-06T11:32:57.217 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:57.221 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:57.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:57.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:57.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:57.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:57.226 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:57.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:57.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:57.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:57.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:57.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:57.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:57.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:57.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:57.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:57.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:57.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:57.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:57.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:57.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:58.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:58.115 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-17 zones={test-zone2, test-zone1} 2021-09-06T11:32:58.115 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-17 zones={test-zone3, test-zone1} 2021-09-06T11:32:58.122 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:58.167 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:58.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:58.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:58.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:58.170 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:58.176 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:58.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:58.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:58.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:58.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:58.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:58.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:58.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:58.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:58.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:58.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:58.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:58.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:58.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:58.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:58.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:59.066 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-17 zones={test-zone3, test-zone1} 2021-09-06T11:32:59.067 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-17 zones={test-zone3, test-zone2} 2021-09-06T11:32:59.072 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:32:59.076 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:59.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:59.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:59.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:59.078 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:32:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:32:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:32:59.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:32:59.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:32:59.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:59.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:59.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:32:59.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:59.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:59.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:32:59.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:59.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:59.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:32:59.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:59.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:59.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:32:59.966 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-17 zones={test-zone3, test-zone2} 2021-09-06T11:32:59.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:59.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:32:59.968 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:00.023 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:00.023 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:00.048 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.048+0000 7f63bb583380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:00.048 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.048+0000 7f63bb583380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:00.049 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.048+0000 7f63bb583380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:00.049 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.048+0000 7f63bb583380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:00.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.094+0000 7f63bb583380 1 bucket sync caught up with source: 2021-09-06T11:33:00.095 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000010.24.5, , 00000000006.19.5, , 00000000010.27.5, , , , , ] 2021-09-06T11:33:00.095 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000010.24.5, , 00000000006.19.5, , 00000000010.27.5, , , , , ] 2021-09-06T11:33:00.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.094+0000 7f63bb583380 0 bucket checkpoint complete 2021-09-06T11:33:00.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.111 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:00.165 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:00.165 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:00.190 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.190+0000 7f854dbff380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:00.191 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.190+0000 7f854dbff380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:00.191 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.190+0000 7f854dbff380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:00.191 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.190+0000 7f854dbff380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:00.222 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.222+0000 7f854dbff380 1 bucket sync caught up with source: 2021-09-06T11:33:00.222 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000010.24.5, , 00000000006.19.5, , 00000000010.27.5, , , , , ] 2021-09-06T11:33:00.223 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000010.24.5, , 00000000006.19.5, , 00000000010.27.5, , , , , ] 2021-09-06T11:33:00.223 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.222+0000 7f854dbff380 0 bucket checkpoint complete 2021-09-06T11:33:00.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.239 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:00.289 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:00.290 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:00.313 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.313+0000 7f78783c3380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:00.313 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.313+0000 7f78783c3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:00.313 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.313+0000 7f78783c3380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:33:00.313 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.313+0000 7f78783c3380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:00.343 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.344+0000 7f78783c3380 1 bucket sync caught up with source: 2021-09-06T11:33:00.344 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000010.24.5, , 00000000006.19.5, , 00000000010.27.5, , , , , ] 2021-09-06T11:33:00.344 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000010.24.5, , 00000000006.19.5, , 00000000010.27.5, , , , , ] 2021-09-06T11:33:00.344 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.344+0000 7f78783c3380 0 bucket checkpoint complete 2021-09-06T11:33:00.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.359 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:00.409 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:00.409 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:00.431 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.431+0000 7fba02359380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:00.431 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.431+0000 7fba02359380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:00.431 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.431+0000 7fba02359380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:00.432 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.431+0000 7fba02359380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:00.461 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.461+0000 7fba02359380 1 bucket sync caught up with source: 2021-09-06T11:33:00.461 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.125.9, , 00000000010.202.5, , 00000000007.165.9, , , , , ] 2021-09-06T11:33:00.461 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.125.9, , 00000000010.202.5, , 00000000007.165.9, , , , , ] 2021-09-06T11:33:00.462 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.461+0000 7fba02359380 0 bucket checkpoint complete 2021-09-06T11:33:00.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.477 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:00.525 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:00.525 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.547+0000 7f7b73049380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:00.548 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.547+0000 7f7b73049380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:00.548 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.547+0000 7f7b73049380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:00.548 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.547+0000 7f7b73049380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:00.579 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.579+0000 7f7b73049380 1 bucket sync caught up with source: 2021-09-06T11:33:00.580 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.125.9, , 00000000010.202.5, , 00000000007.165.9, , , , , ] 2021-09-06T11:33:00.580 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.125.9, , 00000000010.202.5, , 00000000007.165.9, , , , , ] 2021-09-06T11:33:00.580 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.579+0000 7f7b73049380 0 bucket checkpoint complete 2021-09-06T11:33:00.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.596 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:00.645 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:00.646 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:00.669 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.669+0000 7f9dc93b8380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:00.669 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.669+0000 7f9dc93b8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:00.669 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.669+0000 7f9dc93b8380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:00.669 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.669+0000 7f9dc93b8380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:00.699 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.699+0000 7f9dc93b8380 1 bucket sync caught up with source: 2021-09-06T11:33:00.699 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.146.9, , 00000000006.244.5, , 00000000016.208.5, , , , , ] 2021-09-06T11:33:00.699 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.146.9, , 00000000006.244.5, , 00000000016.208.5, , , , , ] 2021-09-06T11:33:00.700 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:00.699+0000 7f9dc93b8380 0 bucket checkpoint complete 2021-09-06T11:33:00.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:00.715 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:00.763 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:00.764 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:00.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.786+0000 7f4c6ecb4380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:00.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.786+0000 7f4c6ecb4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:00.787 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.786+0000 7f4c6ecb4380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:00.787 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.786+0000 7f4c6ecb4380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:00.821 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.822+0000 7f4c6ecb4380 1 bucket sync caught up with source: 2021-09-06T11:33:00.822 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000003.146.9, , 00000000006.244.5, , 00000000016.208.5, , , , , ] 2021-09-06T11:33:00.822 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000003.146.9, , 00000000006.244.5, , 00000000016.208.5, , , , , ] 2021-09-06T11:33:00.822 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:00.822+0000 7f4c6ecb4380 0 bucket checkpoint complete 2021-09-06T11:33:00.841 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-18 zones={test-zone2, test-zone1} 2021-09-06T11:33:00.847 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:00.850 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:00.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:00.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:00.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:00.852 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:00.856 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:00.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:00.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:00.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:00.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:00.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:00.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:00.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:01.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:01.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:01.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:01.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:01.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:01.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:01.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:01.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:01.718 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-18 zones={test-zone2, test-zone1} 2021-09-06T11:33:01.718 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-18 zones={test-zone3, test-zone1} 2021-09-06T11:33:01.724 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:01.728 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:01.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:01.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:01.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:01.729 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:01.733 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:01.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:01.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:01.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:01.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:01.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:01.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:01.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:01.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:02.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:02.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:02.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:02.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:02.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:02.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:02.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:02.602 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-18 zones={test-zone3, test-zone1} 2021-09-06T11:33:02.602 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-18 zones={test-zone3, test-zone2} 2021-09-06T11:33:02.610 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:02.616 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:02.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:02.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:02.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:02.617 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:02.621 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:02.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:02.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:02.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:02.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:02.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:02.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:02.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:02.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:02.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:02.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:03.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:03.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:03.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:03.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:03.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:03.498 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-18 zones={test-zone3, test-zone2} 2021-09-06T11:33:03.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:03.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:03.997 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:04.055 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:04.055 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:04.079 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.079+0000 7f0f9d637380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:04.079 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.079+0000 7f0f9d637380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:04.079 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.079+0000 7f0f9d637380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:04.080 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.079+0000 7f0f9d637380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:04.110 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.110+0000 7f0f9d637380 1 bucket sync caught up with source: 2021-09-06T11:33:04.111 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000011.44.5, , 00000000004.22.5, , 00000000009.23.5, , , , , ] 2021-09-06T11:33:04.111 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000011.44.5, , 00000000004.22.5, , 00000000009.23.5, , , , , ] 2021-09-06T11:33:04.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.110+0000 7f0f9d637380 0 bucket checkpoint complete 2021-09-06T11:33:04.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.127 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:04.183 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:04.183 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:04.206 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.205+0000 7f66b43be380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:04.206 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.205+0000 7f66b43be380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:04.206 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.205+0000 7f66b43be380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:04.206 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.205+0000 7f66b43be380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:04.245 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.245+0000 7f66b43be380 1 bucket sync caught up with source: 2021-09-06T11:33:04.245 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000011.44.5, , 00000000004.22.5, , 00000000009.23.5, , , , , ] 2021-09-06T11:33:04.245 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000011.44.5, , 00000000004.22.5, , 00000000009.23.5, , , , , ] 2021-09-06T11:33:04.245 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.245+0000 7f66b43be380 0 bucket checkpoint complete 2021-09-06T11:33:04.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.261 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:04.310 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:04.310 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:04.332 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.332+0000 7f1c0d41a380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:04.332 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.332+0000 7f1c0d41a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:04.332 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.332+0000 7f1c0d41a380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:33:04.449 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.332+0000 7f1c0d41a380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:04.450 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.367+0000 7f1c0d41a380 1 bucket sync caught up with source: 2021-09-06T11:33:04.450 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000011.44.5, , 00000000004.22.5, , 00000000009.23.5, , , , , ] 2021-09-06T11:33:04.450 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000011.44.5, , 00000000004.22.5, , 00000000009.23.5, , , , , ] 2021-09-06T11:33:04.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.367+0000 7f1c0d41a380 0 bucket checkpoint complete 2021-09-06T11:33:04.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.452 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:04.502 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:04.503 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.526+0000 7f5a7ce50380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.526+0000 7f5a7ce50380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.526+0000 7f5a7ce50380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.526+0000 7f5a7ce50380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:04.560 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.560+0000 7f5a7ce50380 1 bucket sync caught up with source: 2021-09-06T11:33:04.560 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.199.5, , 00000000011.316.5, , 00000000008.138.5, , , , , ] 2021-09-06T11:33:04.561 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.199.5, , 00000000011.316.5, , 00000000008.138.5, , , , , ] 2021-09-06T11:33:04.561 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.560+0000 7f5a7ce50380 0 bucket checkpoint complete 2021-09-06T11:33:04.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.576 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:04.627 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:04.628 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:04.650 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.650+0000 7fd0abdd2380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:04.650 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.650+0000 7fd0abdd2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:04.651 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.650+0000 7fd0abdd2380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:04.651 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.650+0000 7fd0abdd2380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:04.684 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.684+0000 7fd0abdd2380 1 bucket sync caught up with source: 2021-09-06T11:33:04.684 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.199.5, , 00000000011.316.5, , 00000000008.138.5, , , , , ] 2021-09-06T11:33:04.684 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.199.5, , 00000000011.316.5, , 00000000008.138.5, , , , , ] 2021-09-06T11:33:04.685 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.684+0000 7fd0abdd2380 0 bucket checkpoint complete 2021-09-06T11:33:04.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.700 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:04.748 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:04.748 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:04.770 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.770+0000 7f4173be6380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:04.771 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.770+0000 7f4173be6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:04.771 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.770+0000 7f4173be6380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:04.771 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.770+0000 7f4173be6380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:04.800 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.800+0000 7f4173be6380 1 bucket sync caught up with source: 2021-09-06T11:33:04.800 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.126.5, , 00000000004.235.5, , 00000000015.129.5, , , , , ] 2021-09-06T11:33:04.800 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.126.5, , 00000000004.235.5, , 00000000015.129.5, , , , , ] 2021-09-06T11:33:04.800 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:04.800+0000 7f4173be6380 0 bucket checkpoint complete 2021-09-06T11:33:04.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:04.816 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:04.866 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:04.866 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:04.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.888+0000 7fe449aeb380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:04.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.888+0000 7fe449aeb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:04.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.888+0000 7fe449aeb380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:04.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.888+0000 7fe449aeb380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:04.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.924+0000 7fe449aeb380 1 bucket sync caught up with source: 2021-09-06T11:33:04.924 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000004.126.5, , 00000000004.235.5, , 00000000015.129.5, , , , , ] 2021-09-06T11:33:04.924 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000004.126.5, , 00000000004.235.5, , 00000000015.129.5, , , , , ] 2021-09-06T11:33:04.924 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:04.924+0000 7fe449aeb380 0 bucket checkpoint complete 2021-09-06T11:33:04.939 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-16 zones={test-zone2, test-zone1} 2021-09-06T11:33:04.944 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:04.948 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:04.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:04.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:04.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:04.949 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:04.953 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:04.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:04.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:04.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:04.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:04.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:05.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:05.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:05.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:05.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:05.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:05.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:05.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:05.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:05.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:05.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:05.814 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-16 zones={test-zone2, test-zone1} 2021-09-06T11:33:05.814 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-16 zones={test-zone3, test-zone1} 2021-09-06T11:33:05.819 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:05.823 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:05.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:05.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:05.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:05.825 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:05.829 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:05.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:05.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:05.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:05.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:05.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:05.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:05.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:06.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:06.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:06.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:06.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:06.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:06.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:06.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:06.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:06.690 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-16 zones={test-zone3, test-zone1} 2021-09-06T11:33:06.690 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-16 zones={test-zone3, test-zone2} 2021-09-06T11:33:06.696 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:06.700 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:06.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:06.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:06.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:06.702 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:06.706 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:06.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:06.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:06.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:06.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:06.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:06.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:06.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:06.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:06.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:07.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:07.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:07.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:07.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:07.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:07.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:07.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-16 zones={test-zone3, test-zone2} 2021-09-06T11:33:07.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:07.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:07.584 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:07.639 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:07.639 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:07.664 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.665+0000 7fc96cd08380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:07.665 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.665+0000 7fc96cd08380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:07.665 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.665+0000 7fc96cd08380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:07.665 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.665+0000 7fc96cd08380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:07.702 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.703+0000 7fc96cd08380 1 bucket sync caught up with source: 2021-09-06T11:33:07.703 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000011.40.5, , 00000000004.12.5, , 00000000008.46.5, , , , , ] 2021-09-06T11:33:07.703 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000011.40.5, , 00000000004.12.5, , 00000000008.46.5, , , , , ] 2021-09-06T11:33:07.703 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.703+0000 7fc96cd08380 0 bucket checkpoint complete 2021-09-06T11:33:07.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:07.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:07.719 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:07.769 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:07.769 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:07.793 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:07.793+0000 7f7f38cfe380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:07.794 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:07.793+0000 7f7f38cfe380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:07.794 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:07.793+0000 7f7f38cfe380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:07.794 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:07.793+0000 7f7f38cfe380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:07.824 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:07.824+0000 7f7f38cfe380 1 bucket sync caught up with source: 2021-09-06T11:33:07.824 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000011.40.5, , 00000000004.12.5, , 00000000008.46.5, , , , , ] 2021-09-06T11:33:07.824 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000011.40.5, , 00000000004.12.5, , 00000000008.46.5, , , , , ] 2021-09-06T11:33:07.825 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:07.824+0000 7f7f38cfe380 0 bucket checkpoint complete 2021-09-06T11:33:07.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:07.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:07.840 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:07.889 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:07.889 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:07.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.912+0000 7fa364843380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:07.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.912+0000 7fa364843380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:07.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.912+0000 7fa364843380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:33:07.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.912+0000 7fa364843380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:07.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.950+0000 7fa364843380 1 bucket sync caught up with source: 2021-09-06T11:33:07.950 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000011.40.5, , 00000000004.12.5, , 00000000008.46.5, , , , , ] 2021-09-06T11:33:07.950 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000011.40.5, , 00000000004.12.5, , 00000000008.46.5, , , , , ] 2021-09-06T11:33:07.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:07.950+0000 7fa364843380 0 bucket checkpoint complete 2021-09-06T11:33:07.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:07.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:07.965 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:08.013 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:08.013 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.035+0000 7fbe01860380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.035+0000 7fbe01860380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.035+0000 7fbe01860380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:08.036 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.035+0000 7fbe01860380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:08.065 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.065+0000 7fbe01860380 1 bucket sync caught up with source: 2021-09-06T11:33:08.065 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.367.5, , 00000000011.127.5, , 00000000010.201.5, , , , , ] 2021-09-06T11:33:08.065 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.367.5, , 00000000011.127.5, , 00000000010.201.5, , , , , ] 2021-09-06T11:33:08.066 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.065+0000 7fbe01860380 0 bucket checkpoint complete 2021-09-06T11:33:08.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:08.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:08.081 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:08.129 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:08.129 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:08.152 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.152+0000 7f028ba1b380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:08.153 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.152+0000 7f028ba1b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:08.153 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.152+0000 7f028ba1b380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:08.153 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.152+0000 7f028ba1b380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:08.182 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.182+0000 7f028ba1b380 1 bucket sync caught up with source: 2021-09-06T11:33:08.183 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.367.5, , 00000000011.127.5, , 00000000010.201.5, , , , , ] 2021-09-06T11:33:08.183 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.367.5, , 00000000011.127.5, , 00000000010.201.5, , , , , ] 2021-09-06T11:33:08.183 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.182+0000 7f028ba1b380 0 bucket checkpoint complete 2021-09-06T11:33:08.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:08.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:08.198 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:08.269 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.269+0000 7f8143714380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:08.269 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.269+0000 7f8143714380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:08.270 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.269+0000 7f8143714380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:08.270 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.269+0000 7f8143714380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:08.299 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.299+0000 7f8143714380 1 bucket sync caught up with source: 2021-09-06T11:33:08.300 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.154.5, , 00000000004.81.5, , 00000000015.128.5, , , , , ] 2021-09-06T11:33:08.300 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.154.5, , 00000000004.81.5, , 00000000015.128.5, , , , , ] 2021-09-06T11:33:08.300 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:08.299+0000 7f8143714380 0 bucket checkpoint complete 2021-09-06T11:33:08.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:08.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:08.315 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:08.365 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:08.365 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:08.387 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:08.388+0000 7fb875b74380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:08.387 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:08.388+0000 7fb875b74380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:08.388 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:08.388+0000 7fb875b74380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:08.388 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:08.388+0000 7fb875b74380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:08.420 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:08.420+0000 7fb875b74380 1 bucket sync caught up with source: 2021-09-06T11:33:08.420 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000004.154.5, , 00000000004.81.5, , 00000000015.128.5, , , , , ] 2021-09-06T11:33:08.420 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000004.154.5, , 00000000004.81.5, , 00000000015.128.5, , , , , ] 2021-09-06T11:33:08.420 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:08.420+0000 7fb875b74380 0 bucket checkpoint complete 2021-09-06T11:33:08.435 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-17 zones={test-zone2, test-zone1} 2021-09-06T11:33:08.441 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:08.445 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:08.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:08.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:08.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:08.446 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:08.450 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:08.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:08.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:08.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:08.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:08.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:08.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:08.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:08.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:08.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:08.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:08.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:08.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:09.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:09.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:09.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:09.318 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-17 zones={test-zone2, test-zone1} 2021-09-06T11:33:09.318 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-17 zones={test-zone3, test-zone1} 2021-09-06T11:33:09.324 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:09.328 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:09.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:09.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:09.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:09.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:09.333 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:09.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:09.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:09.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:09.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:09.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:09.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:09.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:09.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:09.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:09.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:09.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:09.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:09.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:10.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:10.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:10.194 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-17 zones={test-zone3, test-zone1} 2021-09-06T11:33:10.195 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-17 zones={test-zone3, test-zone2} 2021-09-06T11:33:10.201 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:10.204 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:10.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:10.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:10.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:10.206 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:10.209 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:10.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:10.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:10.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:10.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:10.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:10.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:10.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:10.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:10.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:10.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:10.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:10.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:10.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:10.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:11.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:11.122 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-17 zones={test-zone3, test-zone2} 2021-09-06T11:33:11.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.124 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:11.179 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:11.179 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:11.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.204+0000 7f4bb741e380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:11.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.204+0000 7f4bb741e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:11.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.204+0000 7f4bb741e380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:11.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.204+0000 7f4bb741e380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:11.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.244+0000 7f4bb741e380 1 bucket sync caught up with source: 2021-09-06T11:33:11.244 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000011.30.5, , 00000000007.23.5, , 00000000011.31.5, , , , , ] 2021-09-06T11:33:11.244 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000011.30.5, , 00000000007.23.5, , 00000000011.31.5, , , , , ] 2021-09-06T11:33:11.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.244+0000 7f4bb741e380 0 bucket checkpoint complete 2021-09-06T11:33:11.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.267 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:11.316 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:11.317 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:11.338 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.338+0000 7feaeef2a380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:11.339 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.338+0000 7feaeef2a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:11.339 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.338+0000 7feaeef2a380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:11.339 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.338+0000 7feaeef2a380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:11.368 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.368+0000 7feaeef2a380 1 bucket sync caught up with source: 2021-09-06T11:33:11.368 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000011.30.5, , 00000000007.23.5, , 00000000011.31.5, , , , , ] 2021-09-06T11:33:11.368 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000011.30.5, , 00000000007.23.5, , 00000000011.31.5, , , , , ] 2021-09-06T11:33:11.369 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.368+0000 7feaeef2a380 0 bucket checkpoint complete 2021-09-06T11:33:11.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.384 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:11.433 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:11.433 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:11.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.455+0000 7f7fba40a380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:11.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.455+0000 7f7fba40a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:11.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.455+0000 7f7fba40a380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:33:11.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.455+0000 7f7fba40a380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:11.492 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.492+0000 7f7fba40a380 1 bucket sync caught up with source: 2021-09-06T11:33:11.492 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000011.30.5, , 00000000007.23.5, , 00000000011.31.5, , , , , ] 2021-09-06T11:33:11.493 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000011.30.5, , 00000000007.23.5, , 00000000011.31.5, , , , , ] 2021-09-06T11:33:11.493 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.492+0000 7f7fba40a380 0 bucket checkpoint complete 2021-09-06T11:33:11.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.509 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:11.562 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:11.584 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.584+0000 7f6577834380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:11.584 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.584+0000 7f6577834380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:11.585 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.584+0000 7f6577834380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:11.585 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.584+0000 7f6577834380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:11.617 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.617+0000 7f6577834380 1 bucket sync caught up with source: 2021-09-06T11:33:11.617 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.128.5, , 00000000011.207.5, , 00000000008.168.5, , , , , ] 2021-09-06T11:33:11.618 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.128.5, , 00000000011.207.5, , 00000000008.168.5, , , , , ] 2021-09-06T11:33:11.618 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.617+0000 7f6577834380 0 bucket checkpoint complete 2021-09-06T11:33:11.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.633 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:11.681 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:11.682 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:11.706 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.706+0000 7fbe3b965380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:11.706 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.706+0000 7fbe3b965380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:11.706 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.706+0000 7fbe3b965380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:11.707 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.706+0000 7fbe3b965380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:11.736 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.736+0000 7fbe3b965380 1 bucket sync caught up with source: 2021-09-06T11:33:11.737 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.128.5, , 00000000011.207.5, , 00000000008.168.5, , , , , ] 2021-09-06T11:33:11.737 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.128.5, , 00000000011.207.5, , 00000000008.168.5, , , , , ] 2021-09-06T11:33:11.737 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.736+0000 7fbe3b965380 0 bucket checkpoint complete 2021-09-06T11:33:11.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.752 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:11.801 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:11.801 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:11.824 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.824+0000 7f8f63cb1380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:11.825 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.824+0000 7f8f63cb1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:11.825 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.824+0000 7f8f63cb1380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:11.825 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.824+0000 7f8f63cb1380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.855+0000 7f8f63cb1380 1 bucket sync caught up with source: 2021-09-06T11:33:11.855 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.149.5, , 00000000007.248.5, , 00000000017.211.5, , , , , ] 2021-09-06T11:33:11.855 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.149.5, , 00000000007.248.5, , 00000000017.211.5, , , , , ] 2021-09-06T11:33:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:11.855+0000 7f8f63cb1380 0 bucket checkpoint complete 2021-09-06T11:33:11.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:11.872 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:11.923 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:11.924 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:11.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.947+0000 7faaa0bdb380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:11.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.947+0000 7faaa0bdb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:11.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.947+0000 7faaa0bdb380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:11.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.947+0000 7faaa0bdb380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:11.981 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.982+0000 7faaa0bdb380 1 bucket sync caught up with source: 2021-09-06T11:33:11.981 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, 00000000004.149.5, , 00000000007.248.5, , 00000000017.211.5, , , , , ] 2021-09-06T11:33:11.982 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, 00000000004.149.5, , 00000000007.248.5, , 00000000017.211.5, , , , , ] 2021-09-06T11:33:11.982 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:11.982+0000 7faaa0bdb380 0 bucket checkpoint complete 2021-09-06T11:33:11.996 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-18 zones={test-zone2, test-zone1} 2021-09-06T11:33:12.002 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:12.006 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:12.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:12.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:12.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:12.008 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:12.012 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:12.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:12.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:12.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:12.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:12.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:12.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:12.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:12.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:12.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:12.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:12.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:12.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:12.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:12.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:12.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:12.877 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-18 zones={test-zone2, test-zone1} 2021-09-06T11:33:12.878 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-18 zones={test-zone3, test-zone1} 2021-09-06T11:33:12.883 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:12.887 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:12.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:12.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:12.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:12.888 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:12.892 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:12.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:12.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:12.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:12.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:12.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:12.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:13.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:13.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:13.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:13.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:13.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:13.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:13.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:13.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:13.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:13.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-18 zones={test-zone3, test-zone1} 2021-09-06T11:33:13.762 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-18 zones={test-zone3, test-zone2} 2021-09-06T11:33:13.768 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:13.772 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:13.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:13.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:13.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:13.774 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:13.778 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:13.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T11:33:13.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T11:33:13.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T11:33:13.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:13.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:13.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:13.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:13.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:14.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T11:33:14.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:14.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:14.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T11:33:14.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:14.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:14.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T11:33:14.666 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-18 zones={test-zone3, test-zone2} 2021-09-06T11:33:14.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-09-06T11:33:14.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:14.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:14.691 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:14.732 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:14.732 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:14.779 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:33:14.794 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:33:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:33:14.795 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:33:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:33:14.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927994.690645_100.1", 2021-09-06T11:33:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:14.690645Z" 2021-09-06T11:33:14.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:33:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:33:14.798 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:14.799 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:33:14.800 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:33:14.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:14.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:14.801 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:14.849 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:14.850 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:14.919 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:33:14.920 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:33:14.920 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:33:14.920 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:33:14.920 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:33:14.921 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:33:14.921 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:14.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:14.921 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:33:14.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:14.922 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:33:14.922 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:14.922 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:14.923 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:33:14.923 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:14.923 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:33:14.923 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:14.923 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:33:14.924 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:14.924 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:14.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:14.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:14.925 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:33:14.925 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:14.925 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:14.925 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:33:14.926 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:14.926 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:14.926 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:14.926 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:33:14.926 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:14.927 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:14.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:14.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:14.927 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:33:14.928 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:14.928 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:14.928 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927994.675919_95.1", 2021-09-06T11:33:14.928 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:14.929 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:14.929 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:14.929 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:14.675919Z", 2021-09-06T11:33:14.929 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:14.930 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:14.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:14.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:14.930 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:33:14.931 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:14.931 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:14.931 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:33:14.931 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:14.931 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:14.932 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:14.932 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:33:14.932 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:14.932 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:14.933 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:14.933 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:33:14.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:14.933 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:33:14.933 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:33:14.934 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:33:14.934 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:14.934 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:33:14.935 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630927994.690645_100.1 target=1_1630927994.675919_95.1 2021-09-06T11:33:19.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:19.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:19.938 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:19.986 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:19.986 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:20.053 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:33:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:33:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:33:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:33:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:33:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:33:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:33:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:33:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:33:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:33:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:33:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:33:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:33:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:20.060 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:20.060 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:20.060 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:33:20.060 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:20.061 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:20.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:20.061 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:33:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927994.690645_100.1", 2021-09-06T11:33:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:14.690645Z", 2021-09-06T11:33:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:33:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:33:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:33:20.066 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:20.066 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.066 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.066 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:33:20.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:20.067 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:33:20.067 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:33:20.067 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:33:20.067 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.068 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:33:20.069 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:33:20.069 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:33:20.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:20.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:20.071 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:20.121 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:20.121 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:20.186 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:33:20.186 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:33:20.186 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:33:20.186 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:33:20.187 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:33:20.187 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:33:20.187 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:33:20.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:20.188 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:33:20.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:20.188 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:33:20.188 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:20.188 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:33:20.189 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:33:20.189 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:20.189 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:33:20.189 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:20.190 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:33:20.190 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:33:20.190 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:20.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:20.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:20.191 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:33:20.191 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:20.191 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:33:20.191 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:20.192 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:20.192 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:20.192 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:20.192 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:33:20.192 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:33:20.193 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:20.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:20.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:20.193 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:33:20.194 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:20.194 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:33:20.194 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927994.690645_100.1", 2021-09-06T11:33:20.194 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:20.195 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:20.195 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:20.195 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:14.690645Z", 2021-09-06T11:33:20.195 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:33:20.195 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:20.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:20.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:20.196 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:33:20.196 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:20.196 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:33:20.197 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:33:20.197 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:20.197 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:20.197 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:20.198 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:33:20.198 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:33:20.198 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:20.198 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:20.199 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:33:20.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:20.199 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:33:20.199 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:33:20.199 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:33:20.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:20.200 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:33:20.201 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:33:20.201 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:33:20.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:20.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:20.203 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:20.250 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:20.251 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:20.314 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:33:20.314 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:33:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:33:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:33:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:33:20.315 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:33:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:33:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:20.316 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:33:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:33:20.317 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:33:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:33:20.318 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:33:20.319 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:33:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:20.320 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:33:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:33:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:20.322 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927994.690645_100.1", 2021-09-06T11:33:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:20.323 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:14.690645Z", 2021-09-06T11:33:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:20.324 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:33:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:33:20.325 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:33:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:33:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:20.327 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:33:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:33:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:33:20.328 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:20.328 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:33:20.329 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:33:20.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:20.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:20.376 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:20.426 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:20.427 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:20.448 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:20.448+0000 7f9368e4c380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:20.448 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:20.448+0000 7f9368e4c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:20.449 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:20.448+0000 7f9368e4c380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:20.449 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:20.448+0000 7f9368e4c380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:20.491 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:20.492+0000 7f9368e4c380 1 bucket sync caught up with source: 2021-09-06T11:33:20.491 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , 00000000002.7.10, , , , , ] 2021-09-06T11:33:20.492 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2021-09-06T11:33:20.492 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:20.492+0000 7f9368e4c380 0 bucket checkpoint complete 2021-09-06T11:33:20.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:20.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:20.510 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:20.561 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:20.562 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:20.588 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:20.588+0000 7f30e77dd380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:20.588 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:20.588+0000 7f30e77dd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:20.588 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:20.588+0000 7f30e77dd380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:20.588 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:20.588+0000 7f30e77dd380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:20.620 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:20.620+0000 7f30e77dd380 1 waiting for incremental sync to catch up: 2021-09-06T11:33:20.620 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000002.7.10, , , , , ] 2021-09-06T11:33:20.621 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000003.9.9, , , , , ] 2021-09-06T11:33:25.622 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.622+0000 7f30e77dd380 1 bucket sync caught up with source: 2021-09-06T11:33:25.623 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2021-09-06T11:33:25.623 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000003.9.9, , , , , ] 2021-09-06T11:33:25.623 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.622+0000 7f30e77dd380 0 bucket checkpoint complete 2021-09-06T11:33:25.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:25.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:25.642 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:25.697 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:25.697 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:25.722 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:25.721+0000 7f811b5bd380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:25.722 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:25.721+0000 7f811b5bd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:25.722 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:25.721+0000 7f811b5bd380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:33:25.723 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:25.721+0000 7f811b5bd380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:25.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:25.757+0000 7f811b5bd380 1 bucket sync caught up with source: 2021-09-06T11:33:25.757 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2021-09-06T11:33:25.758 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2021-09-06T11:33:25.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:25.757+0000 7f811b5bd380 0 bucket checkpoint complete 2021-09-06T11:33:25.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:25.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:25.776 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:25.831 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:25.831 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:25.856 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.856+0000 7f7e3b69f380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:25.856 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.856+0000 7f7e3b69f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:25.856 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.856+0000 7f7e3b69f380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:25.856 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.856+0000 7f7e3b69f380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:25.888 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.888+0000 7f7e3b69f380 1 bucket sync caught up with source: 2021-09-06T11:33:25.888 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.107.9, , , , , ] 2021-09-06T11:33:25.888 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.107.9, , , , , ] 2021-09-06T11:33:25.889 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.888+0000 7f7e3b69f380 0 bucket checkpoint complete 2021-09-06T11:33:25.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:25.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:25.904 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:25.953 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:25.953 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:25.976 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.976+0000 7f64b4164380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:25.976 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.976+0000 7f64b4164380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:25.977 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.976+0000 7f64b4164380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:25.977 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:25.976+0000 7f64b4164380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:26.006+0000 7f64b4164380 1 bucket sync caught up with source: 2021-09-06T11:33:26.006 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.107.9, , , , , ] 2021-09-06T11:33:26.006 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.107.9, , , , , ] 2021-09-06T11:33:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:26.006+0000 7f64b4164380 0 bucket checkpoint complete 2021-09-06T11:33:26.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:26.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:26.022 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:26.070 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:26.070 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:26.093 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:26.092+0000 7fb073798380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:26.093 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:26.092+0000 7fb073798380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:26.093 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:26.092+0000 7fb073798380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:26.093 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:26.092+0000 7fb073798380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:26.122 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:26.122+0000 7fb073798380 1 bucket sync caught up with source: 2021-09-06T11:33:26.123 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.167.9, , , , , ] 2021-09-06T11:33:26.123 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.167.9, , , , , ] 2021-09-06T11:33:26.123 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:26.122+0000 7fb073798380 0 bucket checkpoint complete 2021-09-06T11:33:26.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:26.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:26.139 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:26.192 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:26.192 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:26.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:26.214+0000 7fafa9c2d380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:26.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:26.214+0000 7fafa9c2d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:26.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:26.214+0000 7fafa9c2d380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:26.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:26.214+0000 7fafa9c2d380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:26.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:26.247+0000 7fafa9c2d380 1 bucket sync caught up with source: 2021-09-06T11:33:26.247 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , 00000000004.167.9, , , , , ] 2021-09-06T11:33:26.248 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000004.167.9, , , , , ] 2021-09-06T11:33:26.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:26.247+0000 7fafa9c2d380 0 bucket checkpoint complete 2021-09-06T11:33:26.268 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-19 zones={test-zone2, test-zone1} 2021-09-06T11:33:26.275 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:26.319 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:26.320 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:26.323 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:26.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:26.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:26.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:26.486 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-19 zones={test-zone2, test-zone1} 2021-09-06T11:33:26.486 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-19 zones={test-zone3, test-zone1} 2021-09-06T11:33:26.492 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:26.539 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:26.539 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:26.542 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:26.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:26.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:26.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:26.706 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-19 zones={test-zone3, test-zone1} 2021-09-06T11:33:26.706 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-19 zones={test-zone3, test-zone2} 2021-09-06T11:33:26.712 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:26.759 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:26.759 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:26.763 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:26.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:26.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:26.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:26.926 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-19 zones={test-zone3, test-zone2} 2021-09-06T11:33:28.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:28.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:28.336 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:28.391 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:28.392 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:28.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:28.415+0000 7f91f3533380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:28.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:28.415+0000 7f91f3533380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:28.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:28.415+0000 7f91f3533380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:28.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:28.415+0000 7f91f3533380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:28.450 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:28.449+0000 7f91f3533380 1 waiting for incremental sync to catch up: 2021-09-06T11:33:28.450 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , 00000000005.13.9, , , 00000000053.138.8, , ] 2021-09-06T11:33:28.450 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000005.13.9, , , 00000000056.144.11, , ] 2021-09-06T11:33:28.774 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:33:28.774+0000 7fc4b2a0c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T11:33:33.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:33.451+0000 7f91f3533380 1 bucket sync caught up with source: 2021-09-06T11:33:33.452 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , 00000000005.13.9, , , 00000000060.152.9, , ] 2021-09-06T11:33:33.452 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000005.13.9, , , 00000000056.144.11, , ] 2021-09-06T11:33:33.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:33.451+0000 7f91f3533380 0 bucket checkpoint complete 2021-09-06T11:33:33.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:33.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:33.474 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:33.528 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:33.528 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:33.561 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.561+0000 7fa86b69a380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:33.562 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.561+0000 7fa86b69a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:33.562 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.561+0000 7fa86b69a380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:33.562 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.561+0000 7fa86b69a380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:33.593 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.593+0000 7fa86b69a380 1 bucket sync caught up with source: 2021-09-06T11:33:33.593 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000005.13.9, , , 00000000060.152.9, , ] 2021-09-06T11:33:33.593 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000005.13.9, , , 00000000060.152.9, , ] 2021-09-06T11:33:33.593 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.593+0000 7fa86b69a380 0 bucket checkpoint complete 2021-09-06T11:33:33.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:33.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:33.610 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:33.664 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:33.664 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:33.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:33.688+0000 7f9524984380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:33.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:33.688+0000 7f9524984380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:33.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:33.688+0000 7f9524984380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:33:33.690 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:33.688+0000 7f9524984380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:33.726 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:33.726+0000 7f9524984380 1 bucket sync caught up with source: 2021-09-06T11:33:33.727 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , 00000000005.13.9, , , 00000000060.152.9, , ] 2021-09-06T11:33:33.727 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000005.13.9, , , 00000000060.152.9, , ] 2021-09-06T11:33:33.727 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:33.726+0000 7f9524984380 0 bucket checkpoint complete 2021-09-06T11:33:33.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:33.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:33.743 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:33.792 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:33.792 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.815+0000 7f0a7965d380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.815+0000 7f0a7965d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.815+0000 7f0a7965d380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.815+0000 7f0a7965d380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:33.845 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.845+0000 7f0a7965d380 1 bucket sync caught up with source: 2021-09-06T11:33:33.845 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.107.9, , , 00000000056.343.9, , ] 2021-09-06T11:33:33.845 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.107.9, , , 00000000056.343.9, , ] 2021-09-06T11:33:33.845 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.845+0000 7f0a7965d380 0 bucket checkpoint complete 2021-09-06T11:33:33.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:33.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:33.861 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:33.932 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.932+0000 7f85c071e380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.932+0000 7f85c071e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.932+0000 7f85c071e380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:33:33.933 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.932+0000 7f85c071e380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:33.963 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.963+0000 7f85c071e380 1 bucket sync caught up with source: 2021-09-06T11:33:33.964 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.107.9, , , 00000000056.343.9, , ] 2021-09-06T11:33:33.964 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.107.9, , , 00000000056.343.9, , ] 2021-09-06T11:33:33.964 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:33.963+0000 7f85c071e380 0 bucket checkpoint complete 2021-09-06T11:33:33.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:33.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:33.980 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:34.028 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:34.028 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:34.051 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:34.051+0000 7f79e5764380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:34.051 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:34.051+0000 7f79e5764380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:34.051 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:34.051+0000 7f79e5764380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:33:34.051 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:34.051+0000 7f79e5764380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:34.080+0000 7f79e5764380 1 bucket sync caught up with source: 2021-09-06T11:33:34.081 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.167.9, , , 00000000061.277.9, , ] 2021-09-06T11:33:34.081 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.167.9, , , 00000000061.277.9, , ] 2021-09-06T11:33:34.081 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:33:34.080+0000 7f79e5764380 0 bucket checkpoint complete 2021-09-06T11:33:34.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:34.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:33:34.097 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:33:34.148 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:34.148 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:34.172 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:34.171+0000 7f4aafbff380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:33:34.172 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:34.171+0000 7f4aafbff380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:33:34.173 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:34.171+0000 7f4aafbff380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:33:34.173 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:34.171+0000 7f4aafbff380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:33:34.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:34.205+0000 7f4aafbff380 1 bucket sync caught up with source: 2021-09-06T11:33:34.206 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , 00000000004.167.9, , , 00000000061.277.9, , ] 2021-09-06T11:33:34.206 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , 00000000004.167.9, , , 00000000061.277.9, , ] 2021-09-06T11:33:34.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:33:34.205+0000 7f4aafbff380 0 bucket checkpoint complete 2021-09-06T11:33:34.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-19 zones={test-zone2, test-zone1} 2021-09-06T11:33:34.229 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:34.235 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:34.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.249 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:34.254 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:34.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:34.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:34.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:34.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:34.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:34.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:34.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:34.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:34.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:34.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:34.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:34.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:35.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:36.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:37.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:38.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:39.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:40.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:41.990 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-19 zones={test-zone2, test-zone1} 2021-09-06T11:33:41.991 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-19 zones={test-zone3, test-zone1} 2021-09-06T11:33:41.997 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:42.003 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:42.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.017 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:42.023 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:42.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:42.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:42.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:42.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:42.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:42.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:42.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:42.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:42.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:42.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:42.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:42.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:42.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:42.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:42.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:43.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:44.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:45.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:46.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:47.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.665 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:33:48.664+0000 7fc4b0ff6700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T11:33:48.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:48.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:49.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:49.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:49.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:49.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:49.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:49.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:49.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:49.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:49.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:49.778 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-19 zones={test-zone3, test-zone1} 2021-09-06T11:33:49.779 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-19 zones={test-zone3, test-zone2} 2021-09-06T11:33:49.785 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:33:49.790 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:49.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.803 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:33:49.809 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T11:33:49.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:33:49.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:49.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:49.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T11:33:49.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:50.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:51.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:52.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:53.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:54.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:55.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:56.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:57.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:57.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:57.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:57.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:57.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:57.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:33:57.595 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-19 zones={test-zone3, test-zone2} 2021-09-06T11:33:57.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:57.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:57.597 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:57.643 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:57.643 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:57.717 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:33:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:11.472284Z" 2021-09-06T11:33:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:33.475743Z" 2021-09-06T11:33:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:33.477734Z" 2021-09-06T11:33:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:33:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:57.721 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:33:57.722 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-06T11:33:57.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:57.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:57.724 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:57.774 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:57.775 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:57.851 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:33:57.852 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:33:57.852 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:33:57.852 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:33:57.852 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:33:57.853 INFO:teuthology.orchestra.run.smithi139.stdout: "instance_id": 7695315780866180666 2021-09-06T11:33:57.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:57.853 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:33:57.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:57.854 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:33:57.854 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:57.854 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:57.854 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:57.855 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:57.855 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:57.855 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:57.855 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:11.472284Z" 2021-09-06T11:33:57.855 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:57.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:57.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:57.856 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:33:57.856 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:57.857 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:57.857 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:57.857 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:57.857 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:57.857 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:57.858 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:33.475743Z" 2021-09-06T11:33:57.858 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:57.858 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:57.858 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:57.859 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:33:57.859 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:57.859 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:57.859 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:57.859 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:57.860 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:57.860 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:57.860 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:33.477734Z" 2021-09-06T11:33:57.860 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:57.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:57.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:57.861 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:33:57.861 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:57.861 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:57.862 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:33:57.862 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:57.862 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:57.862 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:57.863 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000" 2021-09-06T11:33:57.863 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:57.863 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:57.863 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:33:57.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:57.864 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:33:57.864 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 0, 2021-09-06T11:33:57.864 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 0 2021-09-06T11:33:57.864 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:57.864 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:33:57.866 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-06T11:33:57.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:57.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:57.867 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:57.911 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:57.980 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:33:57.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:57.980 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:57.981 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:11.472284Z" 2021-09-06T11:33:57.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:57.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:57.981 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:57.982 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:33.475743Z" 2021-09-06T11:33:57.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:57.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:57.982 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:57.983 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:33.477734Z" 2021-09-06T11:33:57.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:57.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:57.983 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:57.983 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:33:57.984 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:57.984 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:33:57.985 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-06T11:33:57.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:57.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:57.986 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:58.030 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:58.031 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:58.090 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:33:58.090 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:33:58.090 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:33:58.090 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:33:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:33:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: "instance_id": 8968209698797093946 2021-09-06T11:33:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.091 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:33:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:33:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.092 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.093 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:11.472284Z" 2021-09-06T11:33:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.094 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:33:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.095 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:33.475743Z" 2021-09-06T11:33:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:33:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.098 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:33.477734Z" 2021-09-06T11:33:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:33:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:58.100 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000" 2021-09-06T11:33:58.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:33:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:33:58.102 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 0, 2021-09-06T11:33:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 0 2021-09-06T11:33:58.103 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.103 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:33:58.104 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-06T11:33:58.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.106 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:58.186 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:58.186 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:58.251 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:33:58.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:11.472284Z" 2021-09-06T11:33:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:33.475743Z" 2021-09-06T11:33:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.254 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:58.254 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:33.477734Z" 2021-09-06T11:33:58.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.255 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:58.255 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:33:58.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.255 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:33:58.256 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-06T11:33:58.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.258 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:58.305 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:58.306 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:58.371 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:33:58.371 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:33:58.371 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:33:58.371 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:33:58.372 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:33:58.372 INFO:teuthology.orchestra.run.smithi139.stdout: "instance_id": 11685041783894979537 2021-09-06T11:33:58.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.372 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:33:58.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.373 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:33:58.373 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:58.373 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.373 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:58.374 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:58.374 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:58.374 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:58.374 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:11.472284Z" 2021-09-06T11:33:58.375 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:58.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.375 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:33:58.376 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:58.376 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.376 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:58.376 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:58.376 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:58.377 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:58.377 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:33.475743Z" 2021-09-06T11:33:58.377 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:58.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.378 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.378 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:33:58.378 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:58.378 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.378 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:58.379 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:58.379 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:58.379 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:58.379 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:33.477734Z" 2021-09-06T11:33:58.380 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:58.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.380 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:33:58.381 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:58.381 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.381 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:33:58.381 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:58.381 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:58.382 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:58.382 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000" 2021-09-06T11:33:58.382 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:58.382 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:58.383 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:33:58.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.383 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:33:58.383 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 0, 2021-09-06T11:33:58.384 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 0 2021-09-06T11:33:58.384 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:58.384 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:33:58.385 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-06T11:33:58.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.387 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:58.427 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:58.427 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:58.495 INFO:teuthology.orchestra.run.smithi139.stdout:[ 2021-09-06T11:33:58.495 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.495 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:58.495 INFO:teuthology.orchestra.run.smithi139.stdout: "last_update": "2021-09-06T11:33:11.645350Z" 2021-09-06T11:33:58.496 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.496 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:58.496 INFO:teuthology.orchestra.run.smithi139.stdout: "last_update": "2021-09-06T11:33:33.648551Z" 2021-09-06T11:33:58.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.497 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.497 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:58.497 INFO:teuthology.orchestra.run.smithi139.stdout: "last_update": "2021-09-06T11:33:33.650342Z" 2021-09-06T11:33:58.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.498 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:33:58.498 INFO:teuthology.orchestra.run.smithi139.stdout: "last_update": "0.000000" 2021-09-06T11:33:58.498 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:58.499 INFO:teuthology.orchestra.run.smithi139.stdout:] 2021-09-06T11:33:58.500 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-06T11:33:58.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.501 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:58.548 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:58.549 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:33:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:33:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:33:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:33:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:33:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "instance_id": 13454795610230404625 2021-09-06T11:33:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:33:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:33:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:11.645350Z" 2021-09-06T11:33:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:33:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:33.648551Z" 2021-09-06T11:33:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:33:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:33.650342Z" 2021-09-06T11:33:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:33:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000" 2021-09-06T11:33:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:33:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:33:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 0, 2021-09-06T11:33:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 0 2021-09-06T11:33:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.634 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:33:58.635 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-06T11:33:58.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.637 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:58.678 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:58.678 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:58.746 INFO:teuthology.orchestra.run.smithi139.stdout:[ 2021-09-06T11:33:58.746 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.746 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:58.747 INFO:teuthology.orchestra.run.smithi139.stdout: "last_update": "2021-09-06T11:33:11.645350Z" 2021-09-06T11:33:58.747 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.747 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.747 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:58.748 INFO:teuthology.orchestra.run.smithi139.stdout: "last_update": "2021-09-06T11:33:33.648551Z" 2021-09-06T11:33:58.748 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.748 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.748 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:58.749 INFO:teuthology.orchestra.run.smithi139.stdout: "last_update": "2021-09-06T11:33:33.650342Z" 2021-09-06T11:33:58.749 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:58.749 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:58.749 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:33:58.750 INFO:teuthology.orchestra.run.smithi139.stdout: "last_update": "0.000000" 2021-09-06T11:33:58.750 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:58.750 INFO:teuthology.orchestra.run.smithi139.stdout:] 2021-09-06T11:33:58.751 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-06T11:33:58.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.753 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:58.800 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:58.800 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:58.862 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:33:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:33:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:33:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:33:58.863 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:33:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "instance_id": 11422380928309950340 2021-09-06T11:33:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:33:58.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:33:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:11.645350Z" 2021-09-06T11:33:58.866 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:33:58.867 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.868 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:33.648551Z" 2021-09-06T11:33:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:33:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:58.870 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:33.650342Z" 2021-09-06T11:33:58.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:33:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:58.872 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:58.873 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000" 2021-09-06T11:33:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:33:58.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:33:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 0, 2021-09-06T11:33:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 0 2021-09-06T11:33:58.875 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.876 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:33:58.877 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-06T11:33:58.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.878 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:58.919 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:58.919 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:58.984 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:33:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:11.525651Z" 2021-09-06T11:33:58.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:33.529238Z" 2021-09-06T11:33:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:33.531205Z" 2021-09-06T11:33:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:58.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:33:58.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:58.988 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:33:58.989 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-06T11:33:58.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:58.991 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:59.073 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:59.074 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:59.132 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:33:59.133 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:33:59.133 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:33:59.133 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:33:59.134 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:33:59.134 INFO:teuthology.orchestra.run.smithi094.stdout: "instance_id": 2988449162190220834 2021-09-06T11:33:59.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.134 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:33:59.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.135 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:33:59.135 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:59.135 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:59.136 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:59.136 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:59.136 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:59.136 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:59.137 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:11.525651Z" 2021-09-06T11:33:59.137 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:59.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.138 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:33:59.138 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:59.138 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:59.138 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:59.139 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:59.139 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:59.139 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:59.139 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:33.529238Z" 2021-09-06T11:33:59.140 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:59.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.140 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:33:59.140 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:59.141 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:59.141 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:59.141 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:59.141 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:59.142 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:59.142 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:33.531205Z" 2021-09-06T11:33:59.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:59.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.143 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:33:59.143 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:33:59.143 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "incremental-sync", 2021-09-06T11:33:59.144 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:59.144 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:33:59.144 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:33:59.144 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:33:59.144 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000" 2021-09-06T11:33:59.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:59.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:59.145 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:33:59.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.146 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:33:59.146 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 0, 2021-09-06T11:33:59.146 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 0 2021-09-06T11:33:59.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:59.147 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:33:59.148 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-06T11:33:59.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:59.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:59.150 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:59.228 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:59.229 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:59.294 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:33:59.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.294 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:59.294 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:11.525651Z" 2021-09-06T11:33:59.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.295 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:59.295 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:33.529238Z" 2021-09-06T11:33:59.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.296 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:59.296 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:33.531205Z" 2021-09-06T11:33:59.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.297 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:59.297 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:33:59.297 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:59.298 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:33:59.299 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-06T11:33:59.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:59.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:59.300 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:59.346 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:59.347 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:59.419 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:33:59.419 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:33:59.419 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:33:59.420 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:33:59.420 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:33:59.420 INFO:teuthology.orchestra.run.smithi139.stdout: "instance_id": 8426670124334983876 2021-09-06T11:33:59.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:59.420 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:33:59.421 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:59.421 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:33:59.421 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:59.421 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:59.422 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T11:33:59.422 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:59.422 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:59.422 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:59.423 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:11.525651Z" 2021-09-06T11:33:59.423 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.423 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:59.423 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:59.424 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:33:59.424 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:59.424 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:59.424 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T11:33:59.425 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:59.425 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:59.425 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:59.425 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:33.529238Z" 2021-09-06T11:33:59.426 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.426 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:59.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:59.426 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:33:59.427 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:59.427 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:59.427 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T11:33:59.427 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:59.427 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:59.428 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:59.428 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:33.531205Z" 2021-09-06T11:33:59.428 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:59.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:59.429 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:33:59.429 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:59.429 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "incremental-sync", 2021-09-06T11:33:59.429 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:33:59.430 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:59.430 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:59.430 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:59.430 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000" 2021-09-06T11:33:59.431 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.431 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.431 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:33:59.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:59.432 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:33:59.432 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 0, 2021-09-06T11:33:59.432 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 0 2021-09-06T11:33:59.432 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.432 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:33:59.433 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-06T11:33:59.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-09-06T11:33:59.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:59.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:59.453 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:59.494 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:59.495 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:59.539 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:33:59.558 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:33:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:33:59.558 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:33:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:33:59.559 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:33:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928039.451274_104.1", 2021-09-06T11:33:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:33:59.451274Z" 2021-09-06T11:33:59.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:33:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:33:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:33:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:33:59.561 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:33:59.561 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:33:59.563 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:33:59.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:59.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:33:59.564 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:33:59.609 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:33:59.609 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:33:59.683 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:33:59.683 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:33:59.683 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:33:59.683 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:33:59.684 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:33:59.684 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:33:59.684 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:59.684 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:59.684 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:33:59.685 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:59.685 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:33:59.685 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:59.685 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:59.686 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:33:59.686 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:59.686 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:33:59.686 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:59.686 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:33:59.687 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:59.687 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.687 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:59.687 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:59.688 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:33:59.688 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:59.688 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:59.688 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:33:59.688 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:59.689 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:59.689 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:59.689 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:33:59.689 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:59.690 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.690 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:59.690 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:59.690 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:33:59.691 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:59.691 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:59.691 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928039.443517_101.1", 2021-09-06T11:33:59.691 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:59.691 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:59.692 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:59.692 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:59.443517Z", 2021-09-06T11:33:59.692 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:59.692 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.693 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:59.693 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:33:59.693 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:33:59.693 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:33:59.693 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:33:59.694 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:33:59.694 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:33:59.694 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:33:59.694 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:33:59.695 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:33:59.695 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:33:59.695 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.695 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.696 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:33:59.696 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:33:59.696 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:33:59.696 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:33:59.697 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:33:59.697 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:33:59.697 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:33:59.698 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928039.451274_104.1 target=1_1630928039.443517_101.1 2021-09-06T11:34:04.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:04.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:04.701 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:04.749 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:04.749 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:04.821 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:04.821 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:04.821 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:04.822 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:04.822 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:04.822 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:04.823 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:04.823 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:04.823 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:04.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:04.824 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:04.824 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:04.824 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:04.824 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:04.824 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:04.825 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:04.826 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:04.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:04.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:04.826 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:04.826 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:04.827 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:04.827 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:04.827 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:04.827 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:04.828 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:04.828 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:04.828 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:04.828 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:04.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:04.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:04.829 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:04.829 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:04.829 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:04.829 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928039.451274_104.1", 2021-09-06T11:34:04.830 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:04.830 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:04.830 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:04.830 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:59.451274Z", 2021-09-06T11:34:04.830 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:04.831 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:04.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:04.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:04.831 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:04.831 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:04.832 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:04.832 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:04.832 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:04.832 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:04.833 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:04.833 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:04.833 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:04.833 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:04.833 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:04.834 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:04.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:04.834 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:04.834 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:04.835 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:04.835 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:04.835 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:04.836 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:34:04.836 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:34:04.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:04.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:04.838 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:04.886 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:04.886 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:04.951 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:34:04.952 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:34:04.952 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:34:04.952 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:34:04.953 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:34:04.953 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:04.953 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:04.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:04.954 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:34:04.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:04.954 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:34:04.955 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:04.955 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:04.955 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:04.956 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:04.956 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:34:04.957 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:04.957 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:04.957 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:04.958 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:04.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:04.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:04.959 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:34:04.959 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:04.959 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:04.960 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:34:04.960 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:04.960 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:04.961 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:04.961 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:34:04.961 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:04.962 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:04.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:04.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:04.963 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:34:04.963 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:04.964 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:04.964 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928039.451274_104.1", 2021-09-06T11:34:04.964 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:04.965 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:04.965 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:04.965 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:33:59.451274Z", 2021-09-06T11:34:04.965 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:04.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:04.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:04.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:04.966 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:34:04.967 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:04.967 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:04.967 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:04.967 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:04.968 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:04.968 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:04.968 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:04.968 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:04.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:04.969 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:04.969 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:34:04.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:04.969 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:34:04.969 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:34:04.970 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:34:04.970 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:04.970 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:34:04.971 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:34:04.971 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:34:04.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:04.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:04.973 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:05.018 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:05.018 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:05.085 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:05.085 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:05.085 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:05.086 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:05.086 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:05.086 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:05.086 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:05.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:05.087 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:05.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:05.087 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:05.088 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:05.088 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:05.088 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:05.088 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:05.088 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:05.089 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:05.089 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:05.089 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:05.089 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:05.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:05.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:05.090 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:05.090 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:05.090 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:05.091 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:05.091 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:05.091 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:05.091 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:05.091 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:05.092 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:05.092 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:05.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:05.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:05.092 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:05.093 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:05.093 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:05.093 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928039.451274_104.1", 2021-09-06T11:34:05.093 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:05.093 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:05.094 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:05.094 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:33:59.451274Z", 2021-09-06T11:34:05.094 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:05.094 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:05.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:05.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:05.095 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:05.095 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:05.095 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:05.095 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:05.096 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:05.096 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:05.096 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:05.096 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:05.096 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:05.097 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:05.097 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:05.097 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:05.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:05.098 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:05.098 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:05.098 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:05.098 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:05.099 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:05.100 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:34:05.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:05.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:05.148 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:05.198 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:05.198 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:05.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:05.220+0000 7f1d5dff6380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:05.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:05.220+0000 7f1d5dff6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:05.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:05.220+0000 7f1d5dff6380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:05.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:05.220+0000 7f1d5dff6380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:05.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:05.253+0000 7f1d5dff6380 1 waiting for incremental sync to catch up: 2021-09-06T11:34:05.253 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , , , ] 2021-09-06T11:34:05.254 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000001.8.5, , ] 2021-09-06T11:34:10.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.254+0000 7f1d5dff6380 1 bucket sync caught up with source: 2021-09-06T11:34:10.255 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000001.8.5, , ] 2021-09-06T11:34:10.256 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000001.8.5, , ] 2021-09-06T11:34:10.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.254+0000 7f1d5dff6380 0 bucket checkpoint complete 2021-09-06T11:34:10.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.276 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:10.332 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:10.333 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:10.357 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.357+0000 7f01a78c6380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:10.358 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.357+0000 7f01a78c6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:10.358 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.357+0000 7f01a78c6380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:10.358 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.357+0000 7f01a78c6380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.390+0000 7f01a78c6380 1 bucket sync caught up with source: 2021-09-06T11:34:10.390 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.8.5, , ] 2021-09-06T11:34:10.390 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.8.5, , ] 2021-09-06T11:34:10.390 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.390+0000 7f01a78c6380 0 bucket checkpoint complete 2021-09-06T11:34:10.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.407 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:10.462 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:10.462 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:10.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.486+0000 7f29a60fc380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:10.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.486+0000 7f29a60fc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:10.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.486+0000 7f29a60fc380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:34:10.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.486+0000 7f29a60fc380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:10.519 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.519+0000 7f29a60fc380 1 bucket sync caught up with source: 2021-09-06T11:34:10.519 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000001.8.5, , ] 2021-09-06T11:34:10.520 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000001.8.5, , ] 2021-09-06T11:34:10.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.519+0000 7f29a60fc380 0 bucket checkpoint complete 2021-09-06T11:34:10.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.536 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:10.587 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:10.587 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:10.609 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.609+0000 7f3323453380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:10.610 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.609+0000 7f3323453380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:10.610 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.609+0000 7f3323453380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:10.610 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.609+0000 7f3323453380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:10.645 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.645+0000 7f3323453380 1 bucket sync caught up with source: 2021-09-06T11:34:10.645 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.132.5, , ] 2021-09-06T11:34:10.645 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.132.5, , ] 2021-09-06T11:34:10.645 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.645+0000 7f3323453380 0 bucket checkpoint complete 2021-09-06T11:34:10.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.661 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:10.709 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:10.710 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:10.730 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.730+0000 7fa7841b7380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:10.731 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.730+0000 7fa7841b7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:10.731 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.730+0000 7fa7841b7380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:10.731 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.730+0000 7fa7841b7380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:10.761 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.761+0000 7fa7841b7380 1 bucket sync caught up with source: 2021-09-06T11:34:10.762 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.132.5, , ] 2021-09-06T11:34:10.762 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.132.5, , ] 2021-09-06T11:34:10.762 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.761+0000 7fa7841b7380 0 bucket checkpoint complete 2021-09-06T11:34:10.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.778 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:10.826 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:10.826 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:10.848 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.848+0000 7f5a2ebc7380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:10.848 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.848+0000 7f5a2ebc7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.848+0000 7f5a2ebc7380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.848+0000 7f5a2ebc7380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:10.878 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.878+0000 7f5a2ebc7380 1 bucket sync caught up with source: 2021-09-06T11:34:10.879 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.215.5, , ] 2021-09-06T11:34:10.879 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.215.5, , ] 2021-09-06T11:34:10.879 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:10.878+0000 7f5a2ebc7380 0 bucket checkpoint complete 2021-09-06T11:34:10.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:10.895 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:10.946 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:10.946 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:10.969 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.968+0000 7f30a7017380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:10.969 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.968+0000 7f30a7017380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:10.970 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.968+0000 7f30a7017380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:10.970 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:10.968+0000 7f30a7017380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:11.002 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:11.002+0000 7f30a7017380 1 bucket sync caught up with source: 2021-09-06T11:34:11.003 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000001.215.5, , ] 2021-09-06T11:34:11.003 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000001.215.5, , ] 2021-09-06T11:34:11.003 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:11.002+0000 7f30a7017380 0 bucket checkpoint complete 2021-09-06T11:34:11.019 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-20 zones={test-zone2, test-zone1} 2021-09-06T11:34:11.025 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:11.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:11.071 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:11.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:11.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:11.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:11.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:11.238 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-20 zones={test-zone2, test-zone1} 2021-09-06T11:34:11.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-20 zones={test-zone3, test-zone1} 2021-09-06T11:34:11.243 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:11.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:11.291 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:11.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:11.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:11.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:11.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:11.458 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-20 zones={test-zone3, test-zone1} 2021-09-06T11:34:11.459 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-20 zones={test-zone3, test-zone2} 2021-09-06T11:34:11.464 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:11.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:11.511 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:11.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:11.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:11.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:11.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:11.678 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-20 zones={test-zone3, test-zone2} 2021-09-06T11:34:11.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:11.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:11.686 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:11.727 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:11.728 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:34:11.788 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:34:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:11.788 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:34:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:34:11.789 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:34:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928051.684778_106.1", 2021-09-06T11:34:11.790 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:34:11.684778Z" 2021-09-06T11:34:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:11.791 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:34:11.792 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:11.792 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:34:11.793 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:34:11.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:11.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:11.795 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:11.837 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:11.837 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:11.897 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:11.897 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:11.897 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:11.898 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:11.898 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:11.898 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:11.898 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:11.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:11.899 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:11.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:11.899 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:11.899 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:11.900 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:11.900 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:11.900 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:11.900 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:11.900 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:11.901 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:11.901 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:11.901 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:11.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:11.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:11.902 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:11.902 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:11.902 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:11.902 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:11.903 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:11.903 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:11.903 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:11.903 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:11.903 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:11.904 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:11.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:11.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:11.904 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:11.905 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:11.905 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:11.905 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928051.681225_105.1", 2021-09-06T11:34:11.905 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:11.906 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:11.906 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:11.906 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:11.681225Z", 2021-09-06T11:34:11.906 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:11.906 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:11.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:11.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:11.907 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:11.907 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:11.907 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:11.908 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:11.908 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:11.908 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:11.908 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:11.909 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:11.909 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:11.909 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:11.909 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:11.909 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:11.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:11.910 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:11.910 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:11.910 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:11.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:11.911 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:11.912 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928051.684778_106.1 target=1_1630928051.681225_105.1 2021-09-06T11:34:16.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:16.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:16.914 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:16.963 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:16.964 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:17.044 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:17.044 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:17.045 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:17.045 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:17.045 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:17.045 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:17.046 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:17.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:17.046 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:17.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:17.047 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:17.047 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:17.047 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:17.047 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:17.048 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:17.048 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:17.048 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:17.048 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:17.048 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:17.049 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:17.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:17.049 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:17.050 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:17.050 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:17.050 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:17.050 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:17.051 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:17.051 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:17.051 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:17.051 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:17.051 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:17.052 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:17.052 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:17.052 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:17.052 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:17.053 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928051.684778_106.1", 2021-09-06T11:34:17.053 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:17.053 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:17.053 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:17.053 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:11.684778Z", 2021-09-06T11:34:17.054 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:17.054 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:17.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:17.055 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:17.055 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:17.055 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:17.055 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:17.056 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:17.056 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:17.056 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:17.056 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:17.056 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:17.057 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.057 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.057 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:17.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:17.057 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:17.058 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:17.058 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:17.058 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.058 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:17.060 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:34:17.060 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:34:17.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:17.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:17.062 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:17.110 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:17.175 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:34:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:34:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:34:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:34:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:34:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:34:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:34:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:34:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:17.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:34:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:17.180 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:34:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:34:17.181 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:34:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928051.684778_106.1", 2021-09-06T11:34:17.183 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:34:11.684778Z", 2021-09-06T11:34:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:17.184 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:34:17.185 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:17.186 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:17.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:34:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:34:17.188 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:34:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:34:17.189 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:17.189 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:34:17.190 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:34:17.191 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:34:17.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:17.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:17.192 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:17.235 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:17.235 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:17.307 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:17.307 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:17.307 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:17.308 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:17.308 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:17.308 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:17.309 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:17.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:17.309 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:17.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:17.310 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:17.310 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:17.310 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:17.310 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:17.311 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:17.311 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:17.311 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:17.311 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:17.312 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:17.312 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:17.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:17.312 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:17.313 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:17.313 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:17.313 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:17.313 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:17.313 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:17.314 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:17.314 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:17.314 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:17.314 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.315 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:17.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:17.315 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:17.315 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:17.316 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:17.316 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928051.684778_106.1", 2021-09-06T11:34:17.316 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:17.316 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:17.316 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:17.317 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:11.684778Z", 2021-09-06T11:34:17.317 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:17.317 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:17.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:17.318 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:17.318 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:17.318 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:17.318 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:17.319 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:17.319 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:17.319 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:17.319 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:17.319 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:17.320 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.320 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.320 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:17.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:17.320 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:17.321 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:17.321 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:17.321 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:17.321 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:17.322 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:34:17.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.372 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:17.421 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:17.422 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:17.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.444+0000 7fafb5896380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:17.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.444+0000 7fafb5896380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:17.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.444+0000 7fafb5896380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:17.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.444+0000 7fafb5896380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:17.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.480+0000 7fafb5896380 1 bucket sync caught up with source: 2021-09-06T11:34:17.480 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000003.11.12, , ] 2021-09-06T11:34:17.481 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000003.11.12, , ] 2021-09-06T11:34:17.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.480+0000 7fafb5896380 0 bucket checkpoint complete 2021-09-06T11:34:17.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.499 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:17.549 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:17.550 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:17.572 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.572+0000 7fcd04024380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:17.572 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.572+0000 7fcd04024380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:17.572 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.572+0000 7fcd04024380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:17.573 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.572+0000 7fcd04024380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.604+0000 7fcd04024380 1 bucket sync caught up with source: 2021-09-06T11:34:17.604 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.11.12, , ] 2021-09-06T11:34:17.604 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.11.12, , ] 2021-09-06T11:34:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.604+0000 7fcd04024380 0 bucket checkpoint complete 2021-09-06T11:34:17.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.621 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:17.671 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:17.671 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:17.693 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.692+0000 7f0789571380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:17.693 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.692+0000 7f0789571380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:17.693 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.692+0000 7f0789571380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:34:17.694 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.692+0000 7f0789571380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:17.727 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.727+0000 7f0789571380 1 bucket sync caught up with source: 2021-09-06T11:34:17.728 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000003.11.12, , ] 2021-09-06T11:34:17.728 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000003.11.12, , ] 2021-09-06T11:34:17.728 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:17.727+0000 7f0789571380 0 bucket checkpoint complete 2021-09-06T11:34:17.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.745 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:17.793 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:17.793 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:17.815 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.815+0000 7fba53b1c380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:17.816 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.815+0000 7fba53b1c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:17.816 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.815+0000 7fba53b1c380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:17.816 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.815+0000 7fba53b1c380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:17.846 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.846+0000 7fba53b1c380 1 bucket sync caught up with source: 2021-09-06T11:34:17.847 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000004.147.9, , ] 2021-09-06T11:34:17.847 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000004.147.9, , ] 2021-09-06T11:34:17.847 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.846+0000 7fba53b1c380 0 bucket checkpoint complete 2021-09-06T11:34:17.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.862 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:17.910 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:17.911 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:17.934 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.934+0000 7fca991fb380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:17.934 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.934+0000 7fca991fb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:17.935 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.934+0000 7fca991fb380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:17.935 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.934+0000 7fca991fb380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:17.965 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.965+0000 7fca991fb380 1 bucket sync caught up with source: 2021-09-06T11:34:17.965 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000004.147.9, , ] 2021-09-06T11:34:17.965 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000004.147.9, , ] 2021-09-06T11:34:17.966 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:17.965+0000 7fca991fb380 0 bucket checkpoint complete 2021-09-06T11:34:17.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:17.981 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:18.029 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:18.029 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:18.052+0000 7f371e634380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:18.052+0000 7f371e634380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:18.052 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:18.052+0000 7f371e634380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:18.053 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:18.052+0000 7f371e634380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:18.082 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:18.082+0000 7f371e634380 1 bucket sync caught up with source: 2021-09-06T11:34:18.082 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000004.230.9, , ] 2021-09-06T11:34:18.082 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000004.230.9, , ] 2021-09-06T11:34:18.083 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:18.082+0000 7f371e634380 0 bucket checkpoint complete 2021-09-06T11:34:18.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:18.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:18.098 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:18.147 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:18.147 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:18.173 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:18.172+0000 7fe9c7b10380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:18.173 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:18.172+0000 7fe9c7b10380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:18.173 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:18.172+0000 7fe9c7b10380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:18.173 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:18.172+0000 7fe9c7b10380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:18.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:18.217+0000 7fe9c7b10380 1 bucket sync caught up with source: 2021-09-06T11:34:18.218 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000004.230.9, , ] 2021-09-06T11:34:18.218 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000004.230.9, , ] 2021-09-06T11:34:18.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:18.217+0000 7fe9c7b10380 0 bucket checkpoint complete 2021-09-06T11:34:18.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-20 zones={test-zone2, test-zone1} 2021-09-06T11:34:18.245 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:18.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.292 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:18.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.578 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-20 zones={test-zone2, test-zone1} 2021-09-06T11:34:18.578 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-20 zones={test-zone3, test-zone1} 2021-09-06T11:34:18.584 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:18.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.632 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:18.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:18.918 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-20 zones={test-zone3, test-zone1} 2021-09-06T11:34:18.918 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-20 zones={test-zone3, test-zone2} 2021-09-06T11:34:18.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:18.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.968 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:18.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:18.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:19.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:19.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:19.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:19.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:19.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:19.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:19.254 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-20 zones={test-zone3, test-zone2} 2021-09-06T11:34:19.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:19.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:19.261 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:19.345 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:34:19.360 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:34:19.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:19.361 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:19.361 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:34:19.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:19.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:19.362 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:34:19.362 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:34:19.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:19.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:19.363 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928059.260339_108.1", 2021-09-06T11:34:19.363 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:34:19.260339Z" 2021-09-06T11:34:19.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:19.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:19.364 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:19.364 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:34:19.364 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:19.364 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:34:19.365 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:34:19.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:19.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:19.367 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:19.410 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:19.411 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:19.471 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:19.472 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:19.472 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:19.472 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:19.472 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:19.473 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:19.473 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:19.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:19.473 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:19.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:19.474 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:19.474 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:19.474 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:19.474 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:19.475 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:19.475 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:19.475 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:19.475 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:19.476 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:19.476 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:19.476 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:19.476 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:19.476 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:19.477 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:19.477 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:19.477 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:19.477 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:19.478 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:19.478 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:19.478 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:19.478 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:19.478 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:19.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:19.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:19.479 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:19.479 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:19.480 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:19.480 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928059.257026_107.1", 2021-09-06T11:34:19.480 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:19.480 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:19.480 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:19.481 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:19.257026Z", 2021-09-06T11:34:19.481 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:19.481 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:19.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:19.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:19.482 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:19.482 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:19.482 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:19.483 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:19.483 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:19.483 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:19.483 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:19.483 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:19.484 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:19.484 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:19.484 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:19.484 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:19.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:19.485 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:19.485 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:19.485 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:19.485 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:19.486 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:19.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928059.260339_108.1 target=1_1630928059.257026_107.1 2021-09-06T11:34:24.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:24.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:24.489 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:24.538 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:24.539 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:24.617 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:24.617 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:24.617 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:24.617 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:24.618 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:24.618 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:24.618 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:24.618 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:24.619 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:24.619 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:24.619 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:24.619 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:24.619 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:24.620 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:24.620 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:24.620 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:24.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:24.620 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:24.621 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:24.621 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.621 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:24.621 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:24.621 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:24.622 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:24.622 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:24.622 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:24.622 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:24.623 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:24.623 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:24.623 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:24.623 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:24.623 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.624 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:24.624 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:24.624 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:24.624 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:24.625 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:24.625 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928059.260339_108.1", 2021-09-06T11:34:24.625 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:24.625 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:24.625 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:24.626 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:19.260339Z", 2021-09-06T11:34:24.626 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:24.626 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.626 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:24.627 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:24.627 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:24.627 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:24.627 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:24.627 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:24.628 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:24.628 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:24.628 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:24.628 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:24.628 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:24.629 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.629 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.629 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:24.629 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:24.630 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:24.630 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:24.630 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:24.630 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.630 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:24.631 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:34:24.632 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:34:24.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:24.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:24.633 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:24.684 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:24.684 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:24.750 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:34:24.751 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:34:24.751 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:34:24.751 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:34:24.751 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:34:24.752 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:24.752 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:24.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:24.752 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:34:24.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:24.753 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:34:24.753 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:24.753 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:24.753 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:24.754 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:24.754 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:34:24.754 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:24.754 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:24.755 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:24.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:24.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:24.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:24.756 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:34:24.756 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:24.756 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:24.756 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:34:24.756 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:24.757 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:24.757 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:24.757 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:34:24.757 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:24.758 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:24.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:24.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:24.758 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:34:24.759 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:24.759 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:24.759 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928059.260339_108.1", 2021-09-06T11:34:24.759 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:24.759 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:24.760 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:24.760 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:34:19.260339Z", 2021-09-06T11:34:24.760 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:24.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:24.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:24.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:24.761 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:34:24.761 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:24.761 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:24.762 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:24.762 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:24.762 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:24.762 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:24.763 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:24.763 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:24.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:24.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:24.764 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:34:24.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:24.764 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:34:24.764 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:34:24.764 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:34:24.765 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:24.765 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:34:24.766 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:34:24.766 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:34:24.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:24.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:24.768 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:24.811 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:24.812 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:24.872 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:24.872 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:24.872 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:24.873 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:24.873 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:24.873 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:24.873 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:24.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:24.874 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:24.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:24.874 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:24.874 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:24.875 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:24.875 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:24.875 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:24.875 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:24.876 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:24.876 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:24.876 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:24.876 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:24.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:24.877 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:24.877 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:24.877 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:24.877 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:24.878 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:24.878 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:24.878 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:24.878 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:24.878 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:24.879 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:24.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:24.879 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:24.880 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:24.880 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:24.880 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928059.260339_108.1", 2021-09-06T11:34:24.880 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:24.880 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:24.881 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:24.881 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:19.260339Z", 2021-09-06T11:34:24.881 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:24.881 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:24.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:24.882 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:24.882 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:24.882 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:24.883 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:24.883 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:24.883 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:24.883 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:24.884 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:24.884 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:24.884 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.884 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.884 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:24.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:24.885 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:24.885 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:24.885 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:24.885 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:24.886 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:24.887 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:34:24.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:24.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:24.936 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:24.984 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:24.985 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:25.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.010+0000 7f2996dc2380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:25.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.010+0000 7f2996dc2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:25.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.010+0000 7f2996dc2380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:25.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.010+0000 7f2996dc2380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:25.041 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.041+0000 7f2996dc2380 1 bucket sync caught up with source: 2021-09-06T11:34:25.041 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000005.15.17, , ] 2021-09-06T11:34:25.042 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000005.15.17, , ] 2021-09-06T11:34:25.042 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.041+0000 7f2996dc2380 0 bucket checkpoint complete 2021-09-06T11:34:25.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.059 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:25.111 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:25.111 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:25.133 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.133+0000 7f6fb6e24380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:25.133 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.133+0000 7f6fb6e24380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:25.133 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.133+0000 7f6fb6e24380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:25.134 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.133+0000 7f6fb6e24380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:25.164 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.164+0000 7f6fb6e24380 1 bucket sync caught up with source: 2021-09-06T11:34:25.164 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000005.15.17, , ] 2021-09-06T11:34:25.164 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000005.15.17, , ] 2021-09-06T11:34:25.165 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.164+0000 7f6fb6e24380 0 bucket checkpoint complete 2021-09-06T11:34:25.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.180 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:25.230 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:25.230 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:25.252 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.253+0000 7fd530681380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:25.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.253+0000 7fd530681380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:25.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.253+0000 7fd530681380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:34:25.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.253+0000 7fd530681380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:25.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.292+0000 7fd530681380 1 bucket sync caught up with source: 2021-09-06T11:34:25.292 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000005.15.17, , ] 2021-09-06T11:34:25.292 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000005.15.17, , ] 2021-09-06T11:34:25.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.292+0000 7fd530681380 0 bucket checkpoint complete 2021-09-06T11:34:25.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.308 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:25.357 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:25.357 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.380+0000 7f0c6eb20380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:25.381 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.380+0000 7f0c6eb20380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:25.381 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.380+0000 7f0c6eb20380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:25.381 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.380+0000 7f0c6eb20380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:25.411 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.411+0000 7f0c6eb20380 1 bucket sync caught up with source: 2021-09-06T11:34:25.411 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000007.163.14, , ] 2021-09-06T11:34:25.412 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000007.163.14, , ] 2021-09-06T11:34:25.412 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.411+0000 7f0c6eb20380 0 bucket checkpoint complete 2021-09-06T11:34:25.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.428 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.499+0000 7fae67511380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.499+0000 7fae67511380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.499+0000 7fae67511380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.499+0000 7fae67511380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:25.531 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.531+0000 7fae67511380 1 bucket sync caught up with source: 2021-09-06T11:34:25.531 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000007.163.14, , ] 2021-09-06T11:34:25.531 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000007.163.14, , ] 2021-09-06T11:34:25.531 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.531+0000 7fae67511380 0 bucket checkpoint complete 2021-09-06T11:34:25.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.547 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:25.595 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:25.620 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.620+0000 7f9245634380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:25.620 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.620+0000 7f9245634380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:25.620 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.620+0000 7f9245634380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:25.621 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.620+0000 7f9245634380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:25.650 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.650+0000 7f9245634380 1 bucket sync caught up with source: 2021-09-06T11:34:25.650 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000007.246.14, , ] 2021-09-06T11:34:25.650 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000007.246.14, , ] 2021-09-06T11:34:25.651 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:25.650+0000 7f9245634380 0 bucket checkpoint complete 2021-09-06T11:34:25.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:25.666 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:25.717 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:25.717 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:25.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.740+0000 7f75825b0380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:25.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.740+0000 7f75825b0380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:25.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.740+0000 7f75825b0380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:25.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.740+0000 7f75825b0380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:25.771 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.771+0000 7f75825b0380 1 bucket sync caught up with source: 2021-09-06T11:34:25.771 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000007.246.14, , ] 2021-09-06T11:34:25.771 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000007.246.14, , ] 2021-09-06T11:34:25.771 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:25.771+0000 7f75825b0380 0 bucket checkpoint complete 2021-09-06T11:34:25.787 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-20 zones={test-zone2, test-zone1} 2021-09-06T11:34:25.793 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:25.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:25.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:25.844 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:25.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:25.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:25.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:25.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:25.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.130 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-20 zones={test-zone2, test-zone1} 2021-09-06T11:34:26.131 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-20 zones={test-zone3, test-zone1} 2021-09-06T11:34:26.136 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:26.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:26.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:26.184 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:26.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:26.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:26.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.477 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-20 zones={test-zone3, test-zone1} 2021-09-06T11:34:26.478 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-20 zones={test-zone3, test-zone2} 2021-09-06T11:34:26.483 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:26.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:26.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:26.528 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:26.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:26.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:26.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:26.818 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-20 zones={test-zone3, test-zone2} 2021-09-06T11:34:26.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-09-06T11:34:26.819 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-21 2021-09-06T11:34:26.837 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-22 2021-09-06T11:34:26.874 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-23 2021-09-06T11:34:26.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:26.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:26.929 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:26.969 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:26.969 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:27.021 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:34:27.036 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:34:27.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:27.036 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:27.037 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:34:27.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:27.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:27.037 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:34:27.038 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:34:27.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:27.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:27.038 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928066.925495_126.1", 2021-09-06T11:34:27.039 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:34:26.925495Z" 2021-09-06T11:34:27.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:27.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:27.039 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:27.040 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:34:27.040 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:27.040 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:34:27.041 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:34:27.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:27.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:27.043 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:27.085 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:27.085 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:27.161 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:27.162 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:27.162 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:27.162 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:27.163 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:27.163 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:27.163 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:27.163 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:27.163 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:27.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:27.164 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:27.164 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:27.164 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:27.164 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:27.165 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:27.165 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:27.165 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:27.165 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:27.166 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:27.166 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:27.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:27.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:27.167 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:27.167 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:27.167 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:27.167 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:27.168 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:27.168 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:27.168 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:27.169 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:27.169 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:27.169 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:27.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:27.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:27.170 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:27.170 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:27.170 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:27.171 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928066.827847_109.1", 2021-09-06T11:34:27.171 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:27.172 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:27.172 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:27.172 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:26.827847Z", 2021-09-06T11:34:27.172 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:27.173 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:27.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:27.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:27.173 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:27.173 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:27.174 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:27.174 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:27.174 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:27.174 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:27.175 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:27.175 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:27.175 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:27.175 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:27.176 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:27.176 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:27.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:27.176 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:27.177 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:27.177 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:27.177 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:27.177 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:27.179 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928066.925495_126.1 target=1_1630928066.827847_109.1 2021-09-06T11:34:32.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:32.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:32.182 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:32.230 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:32.230 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:32.309 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:32.309 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:32.310 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:32.310 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:32.310 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:32.310 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:32.311 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:32.311 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:32.311 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:32.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:32.312 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:32.312 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:32.312 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:32.312 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:32.312 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:32.313 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:32.313 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:32.313 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:32.313 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:32.314 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.314 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:32.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:32.314 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:32.315 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:32.315 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:32.315 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:32.315 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:32.315 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:32.316 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:32.316 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:32.316 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:32.316 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:32.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:32.317 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:32.317 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:32.318 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:32.318 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928066.925495_126.1", 2021-09-06T11:34:32.318 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:32.318 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:32.319 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:32.319 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:26.925495Z", 2021-09-06T11:34:32.319 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:32.319 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:32.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:32.320 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:32.320 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:32.320 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:32.321 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:32.321 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:32.321 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:32.321 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:32.321 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:32.322 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:32.322 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.322 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.323 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:32.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:32.323 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:32.323 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:32.323 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:32.324 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.324 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:32.325 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:34:32.326 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:34:32.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:32.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:32.327 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:32.375 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:32.376 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:32.443 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:34:32.443 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:34:32.443 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:34:32.443 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:34:32.444 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:34:32.444 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:32.444 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:32.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:32.445 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:34:32.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:32.445 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:34:32.445 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:32.446 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:32.446 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:32.446 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:32.446 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:34:32.446 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:32.447 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:32.447 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:32.447 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:32.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:32.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:32.448 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:34:32.448 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:32.448 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:32.449 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:34:32.449 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:32.449 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:32.449 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:32.449 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:34:32.450 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:32.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:32.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:32.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:32.451 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:34:32.451 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:32.451 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:32.451 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928066.925495_126.1", 2021-09-06T11:34:32.452 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:32.452 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:32.452 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:32.452 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:34:26.925495Z", 2021-09-06T11:34:32.453 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:32.453 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:32.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:32.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:32.453 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:34:32.454 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:32.454 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:32.454 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:32.454 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:32.455 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:32.455 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:32.455 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:32.455 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:32.456 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:32.456 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:32.456 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:34:32.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:32.456 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:34:32.457 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:34:32.457 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:34:32.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:32.457 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:34:32.458 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:34:32.459 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:34:32.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:32.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:32.461 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:32.503 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:32.504 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:32.564 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:32.565 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:32.565 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:32.565 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:32.565 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:32.566 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:32.566 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:32.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:32.566 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:32.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:32.567 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:32.567 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:32.567 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:32.567 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:32.568 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:32.568 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:32.568 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:32.568 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:32.569 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:32.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:32.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:32.570 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:32.570 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:32.570 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:32.570 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:32.570 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:32.571 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:32.571 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:32.571 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:32.571 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:32.572 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.572 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:32.572 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:32.572 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:32.573 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:32.573 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:32.573 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928066.925495_126.1", 2021-09-06T11:34:32.573 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:32.574 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:32.574 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:32.574 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:26.925495Z", 2021-09-06T11:34:32.574 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:32.575 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.575 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:32.575 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:32.575 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:32.575 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:32.576 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:32.576 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:32.576 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:32.576 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:32.577 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:32.577 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:32.577 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:32.577 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.577 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.578 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:32.578 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:32.578 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:32.578 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:32.579 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:32.579 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:32.579 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:32.580 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:34:32.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:32.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:32.763 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:32.814 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:32.814 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:32.837 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:32.837+0000 7f4141a4a380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:32.837 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:32.837+0000 7f4141a4a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:32.837 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:32.837+0000 7f4141a4a380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:32.838 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:32.837+0000 7f4141a4a380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:32.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:32.878+0000 7f4141a4a380 1 bucket sync caught up with source: 2021-09-06T11:34:32.878 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000003.30.11, , ] 2021-09-06T11:34:32.878 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000003.30.11, , ] 2021-09-06T11:34:32.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:32.878+0000 7f4141a4a380 0 bucket checkpoint complete 2021-09-06T11:34:32.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:32.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:32.896 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:32.950 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:32.951 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:32.980 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:32.980+0000 7f826932f380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:32.980 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:32.980+0000 7f826932f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:32.980 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:32.980+0000 7f826932f380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:32.981 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:32.980+0000 7f826932f380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.014+0000 7f826932f380 1 bucket sync caught up with source: 2021-09-06T11:34:33.014 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.30.11, , ] 2021-09-06T11:34:33.015 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.30.11, , ] 2021-09-06T11:34:33.015 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.014+0000 7f826932f380 0 bucket checkpoint complete 2021-09-06T11:34:33.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:33.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:33.031 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:33.085 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:33.085 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:33.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.111+0000 7f7714b88380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:33.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.111+0000 7f7714b88380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:33.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.111+0000 7f7714b88380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:34:33.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.111+0000 7f7714b88380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:33.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.146+0000 7f7714b88380 1 bucket sync caught up with source: 2021-09-06T11:34:33.146 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000003.30.11, , ] 2021-09-06T11:34:33.146 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000003.30.11, , ] 2021-09-06T11:34:33.146 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.146+0000 7f7714b88380 0 bucket checkpoint complete 2021-09-06T11:34:33.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:33.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:33.161 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:33.213 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:33.213 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.236+0000 7ff42546a380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.236+0000 7ff42546a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.236+0000 7ff42546a380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:33.237 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.236+0000 7ff42546a380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:33.268 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.268+0000 7ff42546a380 1 bucket sync caught up with source: 2021-09-06T11:34:33.268 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.222.5, , ] 2021-09-06T11:34:33.268 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.222.5, , ] 2021-09-06T11:34:33.269 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.268+0000 7ff42546a380 0 bucket checkpoint complete 2021-09-06T11:34:33.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:33.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:33.284 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:33.332 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:33.332 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:33.354 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.354+0000 7f62333c6380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:33.355 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.354+0000 7f62333c6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:33.355 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.354+0000 7f62333c6380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:33.355 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.354+0000 7f62333c6380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:33.385 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.384+0000 7f62333c6380 1 bucket sync caught up with source: 2021-09-06T11:34:33.385 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.222.5, , ] 2021-09-06T11:34:33.385 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.222.5, , ] 2021-09-06T11:34:33.385 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.384+0000 7f62333c6380 0 bucket checkpoint complete 2021-09-06T11:34:33.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:33.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:33.401 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:33.449 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:33.449 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:33.471 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.471+0000 7ff4326c6380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:33.471 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.471+0000 7ff4326c6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:33.472 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.471+0000 7ff4326c6380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:33.472 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.471+0000 7ff4326c6380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:33.502 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.502+0000 7ff4326c6380 1 bucket sync caught up with source: 2021-09-06T11:34:33.502 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.214.11, , ] 2021-09-06T11:34:33.502 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.214.11, , ] 2021-09-06T11:34:33.503 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:33.502+0000 7ff4326c6380 0 bucket checkpoint complete 2021-09-06T11:34:33.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:33.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:33.518 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:33.567 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:33.567 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:33.590 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.591+0000 7f524d9a9380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:33.591 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.591+0000 7f524d9a9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:33.591 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.591+0000 7f524d9a9380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:33.591 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.591+0000 7f524d9a9380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:33.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.624+0000 7f524d9a9380 1 bucket sync caught up with source: 2021-09-06T11:34:33.625 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000003.214.11, , ] 2021-09-06T11:34:33.625 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000003.214.11, , ] 2021-09-06T11:34:33.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:33.624+0000 7f524d9a9380 0 bucket checkpoint complete 2021-09-06T11:34:33.643 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-21 zones={test-zone2, test-zone1} 2021-09-06T11:34:33.649 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:33.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:33.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:33.696 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:33.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:33.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:33.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:33.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:33.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:33.822 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-21 zones={test-zone2, test-zone1} 2021-09-06T11:34:33.822 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-21 zones={test-zone3, test-zone1} 2021-09-06T11:34:33.828 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:33.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:33.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:33.872 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:33.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:33.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:33.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:33.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:33.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:33.998 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-21 zones={test-zone3, test-zone1} 2021-09-06T11:34:33.998 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-21 zones={test-zone3, test-zone2} 2021-09-06T11:34:34.004 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:34.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:34.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:34.052 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:34.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:34.056 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:34.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:34.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:34.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:34.178 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-21 zones={test-zone3, test-zone2} 2021-09-06T11:34:34.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.179 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:34.228 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:34.228 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:34.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.253+0000 7f82b5f88380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:34.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.253+0000 7f82b5f88380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:34.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.253+0000 7f82b5f88380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:34.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.253+0000 7f82b5f88380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:34.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.286+0000 7f82b5f88380 1 bucket sync caught up with source: 2021-09-06T11:34:34.286 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000002.44.5, , ] 2021-09-06T11:34:34.286 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000002.44.5, , ] 2021-09-06T11:34:34.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.286+0000 7f82b5f88380 0 bucket checkpoint complete 2021-09-06T11:34:34.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.303 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:34.352 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:34.352 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:34.375 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.375+0000 7f5a08112380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:34.376 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.375+0000 7f5a08112380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:34.376 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.375+0000 7f5a08112380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:34.376 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.375+0000 7f5a08112380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:34.406 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.406+0000 7f5a08112380 1 bucket sync caught up with source: 2021-09-06T11:34:34.406 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.44.5, , ] 2021-09-06T11:34:34.406 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.44.5, , ] 2021-09-06T11:34:34.407 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.406+0000 7f5a08112380 0 bucket checkpoint complete 2021-09-06T11:34:34.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.422 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:34.471 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:34.471 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:34.493 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.493+0000 7f340aa15380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:34.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.493+0000 7f340aa15380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:34.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.493+0000 7f340aa15380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:34:34.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.493+0000 7f340aa15380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:34.534 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.534+0000 7f340aa15380 1 bucket sync caught up with source: 2021-09-06T11:34:34.534 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000002.44.5, , ] 2021-09-06T11:34:34.534 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000002.44.5, , ] 2021-09-06T11:34:34.534 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.534+0000 7f340aa15380 0 bucket checkpoint complete 2021-09-06T11:34:34.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.551 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:34.601 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:34.601 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:34.626 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.626+0000 7f791211a380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:34.626 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.626+0000 7f791211a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:34.627 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.626+0000 7f791211a380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:34.627 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.626+0000 7f791211a380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:34.658 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.658+0000 7f791211a380 1 bucket sync caught up with source: 2021-09-06T11:34:34.658 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.413.11, , ] 2021-09-06T11:34:34.658 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.413.11, , ] 2021-09-06T11:34:34.659 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.658+0000 7f791211a380 0 bucket checkpoint complete 2021-09-06T11:34:34.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.674 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:34.744 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.744+0000 7f5046974380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:34.744 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.744+0000 7f5046974380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:34.745 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.744+0000 7f5046974380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:34.745 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.744+0000 7f5046974380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:34.774 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.774+0000 7f5046974380 1 bucket sync caught up with source: 2021-09-06T11:34:34.775 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.413.11, , ] 2021-09-06T11:34:34.775 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.413.11, , ] 2021-09-06T11:34:34.775 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.774+0000 7f5046974380 0 bucket checkpoint complete 2021-09-06T11:34:34.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.791 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:34.840 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:34.840 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:34.862 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.862+0000 7f4f0c196380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:34.862 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.862+0000 7f4f0c196380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:34.862 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.862+0000 7f4f0c196380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:34.862 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.862+0000 7f4f0c196380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:34.891 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.891+0000 7f4f0c196380 1 bucket sync caught up with source: 2021-09-06T11:34:34.892 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.197.5, , ] 2021-09-06T11:34:34.892 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.197.5, , ] 2021-09-06T11:34:34.892 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:34.891+0000 7f4f0c196380 0 bucket checkpoint complete 2021-09-06T11:34:34.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:34.908 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:34.958 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:34.958 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:34.980 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.980+0000 7f8b4349f380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:34.980 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.981+0000 7f8b4349f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:34.981 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.981+0000 7f8b4349f380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:34.981 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:34.981+0000 7f8b4349f380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:35.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.014+0000 7f8b4349f380 1 bucket sync caught up with source: 2021-09-06T11:34:35.013 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000002.197.5, , ] 2021-09-06T11:34:35.014 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000002.197.5, , ] 2021-09-06T11:34:35.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.014+0000 7f8b4349f380 0 bucket checkpoint complete 2021-09-06T11:34:35.029 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-22 zones={test-zone2, test-zone1} 2021-09-06T11:34:35.035 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:35.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.080 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:35.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:35.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:35.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:35.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-22 zones={test-zone2, test-zone1} 2021-09-06T11:34:35.206 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-22 zones={test-zone3, test-zone1} 2021-09-06T11:34:35.212 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:35.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.260 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:35.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:35.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:35.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:35.386 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-22 zones={test-zone3, test-zone1} 2021-09-06T11:34:35.386 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-22 zones={test-zone3, test-zone2} 2021-09-06T11:34:35.392 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:35.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.440 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:35.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:35.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:35.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:35.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:35.561 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-22 zones={test-zone3, test-zone2} 2021-09-06T11:34:35.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:35.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:35.563 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:35.612 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:35.612 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:35.634 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.634+0000 7fab6a207380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:35.634 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.634+0000 7fab6a207380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:35.635 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.634+0000 7fab6a207380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:35.635 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.634+0000 7fab6a207380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:35.666 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.667+0000 7fab6a207380 1 bucket sync caught up with source: 2021-09-06T11:34:35.667 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000002.7.5, , ] 2021-09-06T11:34:35.667 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000002.7.5, , ] 2021-09-06T11:34:35.667 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.667+0000 7fab6a207380 0 bucket checkpoint complete 2021-09-06T11:34:35.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:35.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:35.684 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:35.755 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:35.755+0000 7fa1d62a6380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:35.756 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:35.755+0000 7fa1d62a6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:35.756 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:35.755+0000 7fa1d62a6380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:35.756 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:35.755+0000 7fa1d62a6380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:35.785 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:35.785+0000 7fa1d62a6380 1 bucket sync caught up with source: 2021-09-06T11:34:35.785 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.7.5, , ] 2021-09-06T11:34:35.786 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.7.5, , ] 2021-09-06T11:34:35.786 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:35.785+0000 7fa1d62a6380 0 bucket checkpoint complete 2021-09-06T11:34:35.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:35.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:35.801 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:35.850 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:35.850 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:35.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.873+0000 7f40cc864380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:35.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.873+0000 7f40cc864380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:35.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.873+0000 7f40cc864380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:34:35.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.873+0000 7f40cc864380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:35.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.909+0000 7f40cc864380 1 bucket sync caught up with source: 2021-09-06T11:34:35.909 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000002.7.5, , ] 2021-09-06T11:34:35.909 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000002.7.5, , ] 2021-09-06T11:34:35.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:35.909+0000 7f40cc864380 0 bucket checkpoint complete 2021-09-06T11:34:35.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:35.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:35.924 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:35.973 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:35.973 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:35.996 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:35.995+0000 7f2daad71380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:35.996 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:35.995+0000 7f2daad71380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:35.996 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:35.995+0000 7f2daad71380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:35.997 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:35.995+0000 7f2daad71380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:36.026 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.026+0000 7f2daad71380 1 bucket sync caught up with source: 2021-09-06T11:34:36.027 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.134.5, , ] 2021-09-06T11:34:36.027 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.134.5, , ] 2021-09-06T11:34:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.026+0000 7f2daad71380 0 bucket checkpoint complete 2021-09-06T11:34:36.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:36.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:36.042 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:36.090 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:36.112 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.112+0000 7feb1ad69380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:36.113 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.112+0000 7feb1ad69380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:36.113 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.112+0000 7feb1ad69380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:36.113 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.112+0000 7feb1ad69380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:36.142 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.142+0000 7feb1ad69380 1 bucket sync caught up with source: 2021-09-06T11:34:36.142 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.134.5, , ] 2021-09-06T11:34:36.143 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.134.5, , ] 2021-09-06T11:34:36.143 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.142+0000 7feb1ad69380 0 bucket checkpoint complete 2021-09-06T11:34:36.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:36.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:36.158 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:36.206 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:36.206 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:36.228 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.228+0000 7feefc823380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:36.229 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.228+0000 7feefc823380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:36.229 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.228+0000 7feefc823380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:36.229 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.228+0000 7feefc823380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:36.259 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.259+0000 7feefc823380 1 bucket sync caught up with source: 2021-09-06T11:34:36.259 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.177.11, , ] 2021-09-06T11:34:36.260 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.177.11, , ] 2021-09-06T11:34:36.260 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:36.259+0000 7feefc823380 0 bucket checkpoint complete 2021-09-06T11:34:36.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:36.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:36.277 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:36.326 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:36.326 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:36.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:36.349+0000 7ff1d49b9380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:36.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:36.349+0000 7ff1d49b9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:36.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:36.349+0000 7ff1d49b9380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:36.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:36.349+0000 7ff1d49b9380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:36.389 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:36.389+0000 7ff1d49b9380 1 bucket sync caught up with source: 2021-09-06T11:34:36.389 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000003.177.11, , ] 2021-09-06T11:34:36.389 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000003.177.11, , ] 2021-09-06T11:34:36.390 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:36.389+0000 7ff1d49b9380 0 bucket checkpoint complete 2021-09-06T11:34:36.404 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-23 zones={test-zone2, test-zone1} 2021-09-06T11:34:36.411 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:36.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.456 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:36.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:36.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:36.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:36.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-23 zones={test-zone2, test-zone1} 2021-09-06T11:34:36.583 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-23 zones={test-zone3, test-zone1} 2021-09-06T11:34:36.588 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:36.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.636 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:36.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:36.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:36.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:36.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-23 zones={test-zone3, test-zone1} 2021-09-06T11:34:36.762 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-23 zones={test-zone3, test-zone2} 2021-09-06T11:34:36.768 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:36.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.820 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:36.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:36.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:36.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:36.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:36.945 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-23 zones={test-zone3, test-zone2} 2021-09-06T11:34:36.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-09-06T11:34:36.947 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-24 2021-09-06T11:34:36.966 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-25 2021-09-06T11:34:37.002 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-26 2021-09-06T11:34:37.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:37.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:37.085 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:37.125 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:37.126 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:37.170 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:34:37.186 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:34:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:37.187 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:34:37.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:34:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:34:37.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928077.081169_150.1", 2021-09-06T11:34:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:34:37.081169Z" 2021-09-06T11:34:37.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:34:37.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:37.192 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:34:37.193 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:34:37.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:37.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:37.194 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:37.237 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:37.238 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:37.304 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:37.304 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:37.304 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:37.304 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:37.305 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:37.305 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:37.305 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:37.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:37.306 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:37.306 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:37.306 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:37.306 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:37.307 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:37.307 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:37.307 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:37.307 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:37.307 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:37.308 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:37.308 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:37.308 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:37.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:37.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:37.309 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:37.309 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:37.309 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:37.309 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:37.310 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:37.310 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:37.310 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:37.310 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:37.311 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:37.311 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:37.311 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:37.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:37.311 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:37.312 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:37.312 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:37.312 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928076.956316_127.1", 2021-09-06T11:34:37.312 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:37.312 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:37.313 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:37.313 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:36.956316Z", 2021-09-06T11:34:37.313 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:37.313 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:37.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:37.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:37.314 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:37.314 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:37.314 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:37.315 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:37.315 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:37.315 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:37.315 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:37.315 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:37.316 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:37.316 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:37.316 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:37.316 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:37.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:37.317 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:37.317 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:37.317 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:37.317 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:37.318 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:37.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928077.081169_150.1 target=1_1630928076.956316_127.1 2021-09-06T11:34:42.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:42.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:42.321 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:42.369 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:42.370 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:42.438 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928077.081169_150.1", 2021-09-06T11:34:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:37.081169Z", 2021-09-06T11:34:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:42.448 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:42.448 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:42.448 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:42.448 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:42.448 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:42.449 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:42.449 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:42.449 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:42.449 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:42.449 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.450 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.450 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:42.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:42.450 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:42.451 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:42.451 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:42.451 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.451 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:42.452 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T11:34:42.453 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T11:34:42.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:42.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:42.455 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:42.502 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:42.573 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T11:34:42.573 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": { 2021-09-06T11:34:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "info": { 2021-09-06T11:34:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "status": "sync", 2021-09-06T11:34:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "num_shards": 4, 2021-09-06T11:34:42.574 INFO:teuthology.orchestra.run.smithi094.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "markers": [ 2021-09-06T11:34:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:42.575 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 0, 2021-09-06T11:34:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:42.576 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 1, 2021-09-06T11:34:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:42.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 1, 2021-09-06T11:34:42.578 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:34:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:42.579 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "0.000000", 2021-09-06T11:34:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:42.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 2, 2021-09-06T11:34:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928077.081169_150.1", 2021-09-06T11:34:42.581 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:34:37.081169Z", 2021-09-06T11:34:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:42.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "key": 3, 2021-09-06T11:34:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T11:34:42.583 INFO:teuthology.orchestra.run.smithi094.stdout: "state": 1, 2021-09-06T11:34:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "next_step_marker": "", 2021-09-06T11:34:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "total_entries": 0, 2021-09-06T11:34:42.584 INFO:teuthology.orchestra.run.smithi094.stdout: "pos": 0, 2021-09-06T11:34:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T11:34:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:42.585 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T11:34:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "full_sync": { 2021-09-06T11:34:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "total": 1, 2021-09-06T11:34:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: "complete": 1 2021-09-06T11:34:42.586 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:42.587 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T11:34:42.588 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T11:34:42.588 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T11:34:42.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:42.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:42.590 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:42.635 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:42.635 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:42.699 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:42.699 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:42.699 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:42.700 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:42.700 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:42.700 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:42.700 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:42.700 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:42.701 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:42.701 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:42.701 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:42.701 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:42.702 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:42.702 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:42.702 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:42.702 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:42.703 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:42.703 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:42.703 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:42.703 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.703 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:42.704 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:42.704 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:42.704 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:42.704 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:42.704 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:42.705 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:42.705 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:42.705 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:42.705 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:42.706 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:42.706 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.706 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:42.706 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:42.707 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:42.707 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:42.707 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:42.707 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928077.081169_150.1", 2021-09-06T11:34:42.707 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:42.708 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:42.708 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:42.708 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:37.081169Z", 2021-09-06T11:34:42.708 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:42.709 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.709 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:42.709 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:42.709 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:42.709 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:42.710 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:42.710 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:42.710 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:42.710 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:42.710 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:42.711 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:42.711 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:42.711 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.711 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.711 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:42.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:42.712 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:42.712 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:42.712 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:42.713 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:42.713 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:42.714 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T11:34:42.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:42.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:42.901 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:42.951 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:42.952 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:42.976 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:42.976+0000 7f409ce9e380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:42.976 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:42.976+0000 7f409ce9e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:42.977 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:42.976+0000 7f409ce9e380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:42.977 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:42.976+0000 7f409ce9e380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:43.020 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:43.020+0000 7f409ce9e380 1 bucket sync caught up with source: 2021-09-06T11:34:43.020 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000003.21.6, , ] 2021-09-06T11:34:43.020 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000003.21.6, , ] 2021-09-06T11:34:43.021 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:43.020+0000 7f409ce9e380 0 bucket checkpoint complete 2021-09-06T11:34:43.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:43.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:43.037 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:43.089 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:43.089 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:43.113 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.113+0000 7f7c3d1bf380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:43.114 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.113+0000 7f7c3d1bf380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:43.114 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.113+0000 7f7c3d1bf380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:43.114 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.113+0000 7f7c3d1bf380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:43.144 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.144+0000 7f7c3d1bf380 1 bucket sync caught up with source: 2021-09-06T11:34:43.144 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.21.6, , ] 2021-09-06T11:34:43.144 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.21.6, , ] 2021-09-06T11:34:43.144 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.144+0000 7f7c3d1bf380 0 bucket checkpoint complete 2021-09-06T11:34:43.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:43.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:43.162 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:43.212 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:43.212 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:43.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:43.234+0000 7f69acf96380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:43.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:43.234+0000 7f69acf96380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:43.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:43.234+0000 7f69acf96380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:34:43.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:43.234+0000 7f69acf96380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:43.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:43.265+0000 7f69acf96380 1 bucket sync caught up with source: 2021-09-06T11:34:43.265 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000003.21.6, , ] 2021-09-06T11:34:43.265 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000003.21.6, , ] 2021-09-06T11:34:43.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:43.265+0000 7f69acf96380 0 bucket checkpoint complete 2021-09-06T11:34:43.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:43.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:43.282 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:43.330 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:43.331 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:43.352 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.352+0000 7fd9ff05a380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:43.353 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.352+0000 7fd9ff05a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:43.353 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.352+0000 7fd9ff05a380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:43.353 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.352+0000 7fd9ff05a380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:43.383 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:43.383+0000 7fd9ff05a380 1 waiting for incremental sync to catch up: 2021-09-06T11:34:43.383 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , , , ] 2021-09-06T11:34:43.384 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.203.12, , ] 2021-09-06T11:34:48.385 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.384+0000 7fd9ff05a380 1 bucket sync caught up with source: 2021-09-06T11:34:48.385 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.203.12, , ] 2021-09-06T11:34:48.385 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.203.12, , ] 2021-09-06T11:34:48.386 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.384+0000 7fd9ff05a380 0 bucket checkpoint complete 2021-09-06T11:34:48.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:48.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:48.405 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:48.457 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:48.458 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:48.482 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.482+0000 7fbe45617380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:48.482 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.482+0000 7fbe45617380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:48.482 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.482+0000 7fbe45617380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:48.483 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.482+0000 7fbe45617380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.515+0000 7fbe45617380 1 bucket sync caught up with source: 2021-09-06T11:34:48.515 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.203.12, , ] 2021-09-06T11:34:48.516 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.203.12, , ] 2021-09-06T11:34:48.516 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.515+0000 7fbe45617380 0 bucket checkpoint complete 2021-09-06T11:34:48.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:48.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:48.532 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:48.583 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:48.583 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:48.604 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.604+0000 7f96ecc24380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:48.604 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.604+0000 7f96ecc24380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:48.605 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.604+0000 7f96ecc24380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:48.605 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.604+0000 7f96ecc24380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:48.638 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.638+0000 7f96ecc24380 1 bucket sync caught up with source: 2021-09-06T11:34:48.638 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.149.12, , ] 2021-09-06T11:34:48.638 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.149.12, , ] 2021-09-06T11:34:48.638 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:48.638+0000 7f96ecc24380 0 bucket checkpoint complete 2021-09-06T11:34:48.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:48.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:48.656 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:48.711 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:48.711 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:48.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:48.737+0000 7f9435a58380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:48.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:48.737+0000 7f9435a58380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:48.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:48.737+0000 7f9435a58380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:48.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:48.737+0000 7f9435a58380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:48.777 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:48.778+0000 7f9435a58380 1 bucket sync caught up with source: 2021-09-06T11:34:48.778 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000001.149.12, , ] 2021-09-06T11:34:48.778 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000001.149.12, , ] 2021-09-06T11:34:48.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:48.778+0000 7f9435a58380 0 bucket checkpoint complete 2021-09-06T11:34:48.794 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-24 zones={test-zone2, test-zone1} 2021-09-06T11:34:48.800 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:48.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:48.852 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:48.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:48.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:48.905 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-24 zones={test-zone2, test-zone1} 2021-09-06T11:34:48.905 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-24 zones={test-zone3, test-zone1} 2021-09-06T11:34:48.910 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:48.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:48.956 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:48.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:49.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:49.009 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-24 zones={test-zone3, test-zone1} 2021-09-06T11:34:49.009 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-24 zones={test-zone3, test-zone2} 2021-09-06T11:34:49.015 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:49.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:49.060 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:49.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:49.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:49.117 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-24 zones={test-zone3, test-zone2} 2021-09-06T11:34:49.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.119 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:49.169 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:49.170 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:49.192 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.192+0000 7ffa49b0d380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:49.192 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.192+0000 7ffa49b0d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:49.192 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.192+0000 7ffa49b0d380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:49.193 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.192+0000 7ffa49b0d380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:49.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.223+0000 7ffa49b0d380 1 bucket sync caught up with source: 2021-09-06T11:34:49.223 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T11:34:49.223 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T11:34:49.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.223+0000 7ffa49b0d380 0 bucket checkpoint complete 2021-09-06T11:34:49.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.240 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:49.288 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.310+0000 7fdbcb74b380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.310+0000 7fdbcb74b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.310+0000 7fdbcb74b380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.310+0000 7fdbcb74b380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:49.341 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.341+0000 7fdbcb74b380 1 bucket sync caught up with source: 2021-09-06T11:34:49.342 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T11:34:49.342 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T11:34:49.342 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.341+0000 7fdbcb74b380 0 bucket checkpoint complete 2021-09-06T11:34:49.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.358 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:49.407 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:49.407 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:49.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.429+0000 7f3fd0be1380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:49.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.429+0000 7f3fd0be1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:49.430 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.429+0000 7f3fd0be1380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:34:49.430 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.429+0000 7f3fd0be1380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:49.460 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.460+0000 7f3fd0be1380 1 bucket sync caught up with source: 2021-09-06T11:34:49.460 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T11:34:49.461 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T11:34:49.461 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.460+0000 7f3fd0be1380 0 bucket checkpoint complete 2021-09-06T11:34:49.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.476 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:49.525 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:49.547 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.547+0000 7fba299ea380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:49.547 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.547+0000 7fba299ea380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:49.548 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.547+0000 7fba299ea380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:49.548 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.547+0000 7fba299ea380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:49.577 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.577+0000 7fba299ea380 1 bucket sync caught up with source: 2021-09-06T11:34:49.577 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.238.6, , ] 2021-09-06T11:34:49.578 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.238.6, , ] 2021-09-06T11:34:49.578 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.577+0000 7fba299ea380 0 bucket checkpoint complete 2021-09-06T11:34:49.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.594 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:49.642 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:49.642 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:49.667 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.666+0000 7eff57154380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:49.667 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.666+0000 7eff57154380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:49.667 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.666+0000 7eff57154380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:49.668 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.666+0000 7eff57154380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:49.703 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.703+0000 7eff57154380 1 bucket sync caught up with source: 2021-09-06T11:34:49.703 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.238.6, , ] 2021-09-06T11:34:49.704 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.238.6, , ] 2021-09-06T11:34:49.704 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.703+0000 7eff57154380 0 bucket checkpoint complete 2021-09-06T11:34:49.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.720 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:49.769 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:49.791 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.791+0000 7f8783aeb380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:49.791 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.791+0000 7f8783aeb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:49.792 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.791+0000 7f8783aeb380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:49.792 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.791+0000 7f8783aeb380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:49.822 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.822+0000 7f8783aeb380 1 bucket sync caught up with source: 2021-09-06T11:34:49.822 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.63.12, , ] 2021-09-06T11:34:49.822 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.63.12, , ] 2021-09-06T11:34:49.823 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:49.822+0000 7f8783aeb380 0 bucket checkpoint complete 2021-09-06T11:34:49.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:49.838 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:49.889 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:49.889 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:49.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.912+0000 7f89eb99d380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:49.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.912+0000 7f89eb99d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:49.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.912+0000 7f89eb99d380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:49.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.912+0000 7f89eb99d380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:49.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.954+0000 7f89eb99d380 1 bucket sync caught up with source: 2021-09-06T11:34:49.954 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000001.63.12, , ] 2021-09-06T11:34:49.954 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000001.63.12, , ] 2021-09-06T11:34:49.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:49.954+0000 7f89eb99d380 0 bucket checkpoint complete 2021-09-06T11:34:49.972 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-25 zones={test-zone2, test-zone1} 2021-09-06T11:34:49.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:50.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:50.023 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:50.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:50.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:50.077 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-25 zones={test-zone2, test-zone1} 2021-09-06T11:34:50.078 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-25 zones={test-zone3, test-zone1} 2021-09-06T11:34:50.083 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:50.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:50.127 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:50.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:50.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:50.181 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-25 zones={test-zone3, test-zone1} 2021-09-06T11:34:50.182 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-25 zones={test-zone3, test-zone2} 2021-09-06T11:34:50.187 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:50.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:50.235 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:50.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:50.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:50.290 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-25 zones={test-zone3, test-zone2} 2021-09-06T11:34:50.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.293 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:50.345 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:50.346 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:50.369 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.369+0000 7f3c5f34d380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:50.369 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.369+0000 7f3c5f34d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:50.369 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.369+0000 7f3c5f34d380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:50.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.369+0000 7f3c5f34d380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:50.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.402+0000 7f3c5f34d380 1 bucket sync caught up with source: 2021-09-06T11:34:50.402 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-09-06T11:34:50.402 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-09-06T11:34:50.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.402+0000 7f3c5f34d380 0 bucket checkpoint complete 2021-09-06T11:34:50.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.419 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:50.468 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:50.468 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:50.491 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.491+0000 7f4b3568c380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:50.491 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.491+0000 7f4b3568c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:50.491 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.491+0000 7f4b3568c380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:50.491 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.491+0000 7f4b3568c380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.522+0000 7f4b3568c380 1 bucket sync caught up with source: 2021-09-06T11:34:50.522 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-09-06T11:34:50.522 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-09-06T11:34:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.522+0000 7f4b3568c380 0 bucket checkpoint complete 2021-09-06T11:34:50.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.538 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:50.588 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:50.589 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:50.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.611+0000 7f873f980380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:50.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.611+0000 7f873f980380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:50.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.611+0000 7f873f980380 1 Zone: test-zone4 (4487d356-c18b-4713-bd68-02dfa183d02a) 2021-09-06T11:34:50.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.611+0000 7f873f980380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:50.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.642+0000 7f873f980380 1 bucket sync caught up with source: 2021-09-06T11:34:50.642 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-09-06T11:34:50.643 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-09-06T11:34:50.643 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:50.642+0000 7f873f980380 0 bucket checkpoint complete 2021-09-06T11:34:50.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.658 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:50.708 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.731+0000 7f0e1156e380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.731+0000 7f0e1156e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.731+0000 7f0e1156e380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:50.731 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.731+0000 7f0e1156e380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:50.762 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.762+0000 7f0e1156e380 1 bucket sync caught up with source: 2021-09-06T11:34:50.762 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.292.12, , ] 2021-09-06T11:34:50.762 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.292.12, , ] 2021-09-06T11:34:50.762 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.762+0000 7f0e1156e380 0 bucket checkpoint complete 2021-09-06T11:34:50.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.778 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:50.826 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:50.826 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.848+0000 7f3c8f66e380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:50.849 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.848+0000 7f3c8f66e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:50.849 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.848+0000 7f3c8f66e380 1 Zone: test-zone3 (c248f771-5d9b-48a2-a793-2bc85cc1f251) 2021-09-06T11:34:50.849 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.848+0000 7f3c8f66e380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:50.879 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.879+0000 7f3c8f66e380 1 bucket sync caught up with source: 2021-09-06T11:34:50.879 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.292.12, , ] 2021-09-06T11:34:50.880 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.292.12, , ] 2021-09-06T11:34:50.880 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.879+0000 7f3c8f66e380 0 bucket checkpoint complete 2021-09-06T11:34:50.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:50.896 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ukmzsk-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:50.943 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:50.944 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:50.966 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.966+0000 7f8f1cf92380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:50.966 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.966+0000 7f8f1cf92380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:50.967 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.966+0000 7f8f1cf92380 1 Zone: test-zone1 (7556a536-76a8-4c42-adbe-379e64c0e1a8) 2021-09-06T11:34:50.967 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.966+0000 7f8f1cf92380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:50.996 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.996+0000 7f8f1cf92380 1 bucket sync caught up with source: 2021-09-06T11:34:50.996 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.296.6, , ] 2021-09-06T11:34:50.996 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.296.6, , ] 2021-09-06T11:34:50.996 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T11:34:50.996+0000 7f8f1cf92380 0 bucket checkpoint complete 2021-09-06T11:34:51.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:51.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ukmzsk-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T11:34:51.012 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ukmzsk-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T11:34:51.067 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:51.067 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:51.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:51.093+0000 7fd0f5ba6380 1 Realm: test-realm (33774ddf-9586-4cf3-8e79-39a5aa159120) 2021-09-06T11:34:51.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:51.093+0000 7fd0f5ba6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T11:34:51.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:51.093+0000 7fd0f5ba6380 1 Zone: test-zone2 (2f90b352-42d8-4c14-ad62-fe843301400a) 2021-09-06T11:34:51.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:51.093+0000 7fd0f5ba6380 1 using period configuration: 06970c73-91a9-4240-80e4-aa07d550699b:4 2021-09-06T11:34:51.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:51.140+0000 7fd0f5ba6380 1 bucket sync caught up with source: 2021-09-06T11:34:51.140 INFO:teuthology.orchestra.run.smithi139.stderr: local status: [, , , , , , , , 00000000003.296.6, , ] 2021-09-06T11:34:51.140 INFO:teuthology.orchestra.run.smithi139.stderr: remote markers: [, , , , , , , , 00000000003.296.6, , ] 2021-09-06T11:34:51.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:34:51.140+0000 7fd0f5ba6380 0 bucket checkpoint complete 2021-09-06T11:34:51.156 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-26 zones={test-zone2, test-zone1} 2021-09-06T11:34:51.161 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:51.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:51.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:51.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:51.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:51.262 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-26 zones={test-zone2, test-zone1} 2021-09-06T11:34:51.262 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-26 zones={test-zone3, test-zone1} 2021-09-06T11:34:51.267 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:51.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:51.311 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:51.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:51.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:51.365 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-26 zones={test-zone3, test-zone1} 2021-09-06T11:34:51.366 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ukmzsk-26 zones={test-zone3, test-zone2} 2021-09-06T11:34:51.371 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T11:34:51.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:51.415 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T11:34:51.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T11:34:51.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T11:34:51.470 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ukmzsk-26 zones={test-zone3, test-zone2} 2021-09-06T11:34:51.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-09-06T11:34:51.472 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-27 2021-09-06T11:34:51.490 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-28 2021-09-06T11:34:51.529 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-29 2021-09-06T11:34:51.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2021-09-06T11:34:51.724 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-30 2021-09-06T11:34:51.742 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-31 2021-09-06T11:34:51.782 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-32 2021-09-06T11:34:51.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2021-09-06T11:34:51.984 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-33 2021-09-06T11:34:52.021 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-34 2021-09-06T11:34:52.192 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-35 2021-09-06T11:34:54.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2021-09-06T11:34:54.536 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-36 2021-09-06T11:34:54.555 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-37 2021-09-06T11:34:54.594 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-38 2021-09-06T11:34:54.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:54.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:54.639 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:54.683 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:54.683 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:54.730 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:34:54.745 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:34:54.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:54.746 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:54.747 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:34:54.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:54.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:54.747 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:34:54.747 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:34:54.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:54.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:54.748 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928094.616311_216.1", 2021-09-06T11:34:54.748 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:34:54.616311Z" 2021-09-06T11:34:54.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:34:54.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:34:54.749 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:54.749 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:34:54.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:34:54.750 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:34:54.751 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:34:54.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:54.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:54.753 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:34:54.807 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:34:54.807 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:34:54.932 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:34:54.933 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:34:54.933 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:34:54.933 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:34:54.934 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:34:54.934 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:34:54.934 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:54.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:54.935 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:34:54.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:54.935 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:34:54.935 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:54.936 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:54.936 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:34:54.936 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:54.936 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:34:54.936 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:54.937 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:34:54.937 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:54.937 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:54.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:54.938 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:54.938 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:34:54.938 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:54.938 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:54.938 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:34:54.939 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:54.939 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:54.939 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:54.940 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:34:54.940 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:54.940 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:54.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:54.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:54.941 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:34:54.941 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:54.941 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:54.942 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.547334_162.1", 2021-09-06T11:34:54.942 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:54.942 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:54.942 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:54.942 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.547334Z", 2021-09-06T11:34:54.943 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:54.943 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:54.943 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:54.943 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:34:54.944 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:34:54.944 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:34:54.944 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:34:54.944 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:34:54.944 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:34:54.945 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:34:54.945 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:34:54.945 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:34:54.945 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:34:54.946 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:54.946 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:54.946 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:34:54.946 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:34:54.946 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:34:54.947 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:34:54.947 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:34:54.947 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:34:54.948 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:34:54.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.547334_162.1 2021-09-06T11:34:59.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:59.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:34:59.952 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:00.033 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:00.034 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:00.366 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:00.367 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:00.367 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:00.367 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:00.368 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:00.368 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:00.368 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:00.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:00.369 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:00.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:00.369 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:00.369 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:00.370 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:00.370 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:00.370 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:00.370 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:00.371 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:00.371 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:00.371 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:00.371 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:00.371 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:00.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:00.372 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:00.372 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:00.372 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:00.373 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:00.373 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:00.373 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:00.373 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:00.373 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:00.374 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:00.374 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:00.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:00.374 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:00.375 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:00.375 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:00.375 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:00.375 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:35:00.376 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:00.376 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:00.376 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:00.376 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:35:00.376 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:00.377 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:00.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:00.377 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:00.377 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:00.377 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:00.378 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:00.378 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:00.378 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:00.378 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:00.378 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:00.379 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:00.379 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:00.379 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:00.379 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:00.380 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:00.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:00.380 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:00.380 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:00.381 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:00.381 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:00.381 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:00.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.508454_157.1 2021-09-06T11:35:05.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:05.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:05.386 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:05.454 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:05.454 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:05.539 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:05.539 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:05.540 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:05.541 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:05.542 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:05.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:05.544 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:05.545 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:05.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:05.547 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:35:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:35:05.548 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:05.549 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:05.550 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:05.551 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:05.552 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:05.553 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:05.553 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:05.554 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.508454_157.1 2021-09-06T11:35:10.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:10.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:10.557 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:10.659 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:10.659 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:11.106 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:11.106 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:11.107 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:11.107 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:11.107 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:11.107 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:11.108 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:11.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:11.108 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:11.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:11.108 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:11.109 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:11.109 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:11.109 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:11.109 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:11.110 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:11.110 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:11.110 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:11.110 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:11.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:11.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:11.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:11.111 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:11.111 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:11.112 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:11.112 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:11.112 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:11.112 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:11.113 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:11.113 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:11.113 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:11.113 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:11.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:11.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:11.114 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:11.114 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:11.114 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:11.115 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:35:11.115 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:11.115 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:11.115 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:11.116 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:35:11.116 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:11.116 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:11.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:11.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:11.117 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:11.117 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:11.117 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:11.117 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:11.118 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:11.118 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:11.118 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:11.118 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:11.119 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:11.119 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:11.119 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:11.119 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:11.120 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:11.120 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:11.120 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:11.120 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:11.121 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:11.121 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:11.121 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:35:16.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:16.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:16.124 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:16.167 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:16.168 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:16.234 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:16.235 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:16.235 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:16.235 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:16.235 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:16.236 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:16.236 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:16.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:16.236 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:16.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:16.237 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:16.237 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:16.237 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:16.238 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:16.238 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:16.238 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:16.238 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:16.239 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:16.239 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:16.239 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:16.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:16.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:16.240 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:16.240 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:16.240 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:16.240 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:16.241 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:16.241 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:16.241 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:16.241 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:16.241 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:16.242 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:16.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:16.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:16.242 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:16.243 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:16.243 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:16.243 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:35:16.243 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:16.244 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:16.244 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:16.244 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:35:16.244 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:16.244 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:16.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:16.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:16.245 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:16.245 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:16.246 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:16.246 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:16.246 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:16.246 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:16.246 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:16.247 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:16.247 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:16.247 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:16.247 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:16.247 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:16.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:16.248 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:16.248 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:16.248 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:16.249 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:16.249 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:16.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:35:21.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:21.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:21.253 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:21.298 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:21.298 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:21.366 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:21.366 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:21.366 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:21.367 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:21.367 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:21.367 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:21.367 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:21.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:21.368 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:21.368 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:21.368 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:21.369 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:21.369 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:21.369 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:21.369 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:21.370 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:21.370 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:21.370 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:21.370 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:21.370 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:21.371 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:21.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:21.371 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:21.371 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:21.372 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:21.372 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:21.372 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:21.372 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:21.372 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:21.373 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:21.373 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:21.373 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:21.373 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:21.374 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:21.374 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:21.374 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:21.374 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:21.375 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:35:21.375 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:21.375 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:21.375 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:21.375 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:35:21.376 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:21.376 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:21.376 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:21.376 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:21.377 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:21.377 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:21.377 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:21.377 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:21.378 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:21.378 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:21.378 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:21.378 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:21.379 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:21.379 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:21.379 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:21.379 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:21.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:21.380 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:21.380 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:21.380 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:21.381 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:21.381 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:21.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:35:26.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:26.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:26.385 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:26.453 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:26.453 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:26.564 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:26.564 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:26.565 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:26.565 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:26.565 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:26.565 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:26.566 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:26.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:26.566 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:26.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:26.567 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:26.567 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:26.567 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:26.567 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:26.567 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:26.568 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:26.568 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:26.568 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:26.568 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:26.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:26.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:26.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:26.569 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:26.570 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:26.570 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:26.570 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:26.570 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:26.570 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:26.571 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:26.571 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:26.571 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:26.571 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:26.572 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:26.572 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:26.572 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:26.572 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:26.573 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:26.573 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:35:26.573 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:26.573 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:26.573 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:26.574 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:35:26.574 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:26.574 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:26.574 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:26.575 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:26.575 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:26.575 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:26.575 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:26.575 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:26.576 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:26.576 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:26.576 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:26.576 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:26.577 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:26.577 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:26.577 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:26.577 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:26.578 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:26.578 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:26.578 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:26.578 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:26.578 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:26.579 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:26.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.508454_157.1 2021-09-06T11:35:31.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:31.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:31.582 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:31.657 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:31.657 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:31.879 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:31.880 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:31.880 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:31.880 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:31.881 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:31.881 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:31.881 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:31.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:31.882 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:31.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:31.882 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:31.883 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:31.883 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:31.883 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:31.883 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:31.884 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:31.884 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:31.884 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:31.884 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:31.885 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:31.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:31.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:31.885 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:31.885 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:31.886 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:31.886 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:31.886 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:31.886 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:31.887 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:31.887 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:31.887 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:31.887 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:31.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:31.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:31.888 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:31.888 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:31.889 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:31.889 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:35:31.889 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:31.889 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:31.890 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:31.890 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:35:31.890 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:31.890 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:31.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:31.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:31.891 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:31.891 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:31.892 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:31.892 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:31.892 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:31.892 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:31.892 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:31.893 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:31.893 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:31.893 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:31.893 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:31.894 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:31.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:31.894 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:31.894 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:31.895 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:31.895 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:31.895 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:31.897 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.544174_161.1 2021-09-06T11:35:36.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:36.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:36.900 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:36.952 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:36.952 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:37.147 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:37.147 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:37.147 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:37.148 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:37.148 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:37.148 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:37.148 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:37.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:37.149 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:37.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:37.149 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:37.150 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:37.150 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:37.150 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:37.150 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:37.151 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:37.151 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:37.151 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:37.151 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:37.151 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:37.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:37.152 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:37.152 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:37.152 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:37.153 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:37.153 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:37.153 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:37.153 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:37.154 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:37.154 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:37.154 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:37.154 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:37.155 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:37.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:37.155 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:37.155 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:37.156 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:37.156 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.539819_159.1", 2021-09-06T11:35:37.156 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:37.156 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:37.157 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:37.157 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.539819Z", 2021-09-06T11:35:37.157 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:37.157 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:37.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:37.158 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:37.158 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:37.158 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:37.158 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:37.159 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:37.159 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:37.159 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:37.159 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:37.159 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:37.160 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:37.160 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:37.160 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:37.160 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:37.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:37.161 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:37.161 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:37.161 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:37.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:37.162 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:37.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.539819_159.1 2021-09-06T11:35:42.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:42.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:42.165 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:42.219 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:42.219 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:42.437 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:42.437 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:42.437 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:42.437 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:35:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:35:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:42.448 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:42.448 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:42.448 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:42.448 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:42.449 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:42.449 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:42.449 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:42.449 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:42.449 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:42.450 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:42.450 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:42.450 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:42.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:42.451 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:42.451 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:42.451 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:42.451 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:42.452 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:42.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:35:47.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:47.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:47.455 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:47.501 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:47.502 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:47.724 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:47.726 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:47.727 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:47.728 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:47.729 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:47.730 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:47.731 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:47.732 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:35:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:35:47.733 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:47.734 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:47.735 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:47.736 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:47.737 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:47.738 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:47.739 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:47.741 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.507089_156.1 2021-09-06T11:35:52.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:52.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:52.744 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:52.834 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:52.835 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:52.927 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:52.927 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:52.935 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:52.935 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:52.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:52.935 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:52.936 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:52.936 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:52.936 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:35:52.936 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:52.936 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:52.937 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:52.937 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:35:52.937 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:52.937 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:52.938 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:52.938 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:52.938 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:52.938 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:52.940 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:52.943 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:52.943 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:52.943 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:52.943 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:52.945 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:52.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:52.945 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:52.947 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:35:57.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:57.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:35:57.949 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:35:58.043 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:35:58.043 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:35:58.432 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:35:58.433 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:35:58.433 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:35:58.433 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:35:58.433 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:35:58.434 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:35:58.434 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:58.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:58.434 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:35:58.435 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:58.435 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:35:58.435 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:58.435 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:58.436 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:35:58.436 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:58.436 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:35:58.437 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:58.437 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:35:58.437 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:58.437 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:58.437 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:58.438 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:58.438 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:35:58.438 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:58.438 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:58.439 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:35:58.439 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:58.439 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:58.439 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:58.440 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:35:58.440 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:58.440 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:58.440 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:58.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:58.441 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:35:58.441 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:58.441 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:58.442 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:35:58.442 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:58.442 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:58.442 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:58.443 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:35:58.443 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:58.443 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:58.443 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:58.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:35:58.444 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:35:58.444 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:35:58.444 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:35:58.445 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:35:58.445 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:35:58.445 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:35:58.445 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:35:58.445 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:35:58.446 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:35:58.446 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:58.446 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:58.446 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:35:58.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:35:58.447 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:35:58.447 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:35:58.447 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:35:58.448 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:35:58.448 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:35:58.449 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.484822_152.1 2021-09-06T11:36:03.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:03.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:03.452 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:03.555 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:03.555 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:03.937 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:03.937 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:03.938 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:03.938 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:03.938 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:03.938 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:03.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:03.939 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:03.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:03.939 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:03.940 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:03.940 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:03.940 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:03.940 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:03.941 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:03.941 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:03.941 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:03.941 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:03.942 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:03.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:03.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:03.942 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:03.943 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:03.943 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:03.943 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:03.943 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:03.944 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:03.944 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:03.944 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:03.944 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:03.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:03.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:03.945 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:03.945 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:03.945 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:03.946 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:03.946 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:36:03.946 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:03.946 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:03.947 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:03.947 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:36:03.947 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:03.947 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:03.948 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:03.948 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:03.948 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:03.948 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:03.948 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:03.949 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:03.949 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:03.949 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:03.949 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:03.950 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:03.950 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:03.950 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:03.950 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:03.951 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:03.951 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:03.951 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:03.951 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:03.951 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:03.952 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:03.952 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:03.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:36:08.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:08.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:08.955 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:09.046 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:09.046 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:09.694 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:09.694 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:09.695 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:09.695 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:09.695 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:09.695 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:09.696 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:09.696 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:09.696 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:09.696 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:09.697 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:09.697 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:09.697 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:09.697 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:09.698 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:09.698 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:09.698 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:09.698 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:09.699 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:09.699 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:09.699 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:09.699 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:09.699 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:09.700 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:09.700 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:09.700 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:09.700 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:09.701 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:09.701 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:09.701 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:09.701 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:09.702 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:09.702 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:09.702 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:09.702 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:09.703 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:09.703 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:09.703 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:36:09.703 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:09.703 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:09.704 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:09.704 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:36:09.704 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:09.704 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:09.705 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:09.705 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:09.705 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:09.705 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:09.705 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:09.706 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:09.706 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:09.706 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:09.706 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:09.707 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:09.707 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:09.707 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:09.707 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:09.708 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:09.708 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:09.708 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:09.708 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:09.708 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:09.709 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:09.709 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:09.709 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.508454_157.1 2021-09-06T11:36:14.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:14.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:14.712 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:14.801 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:14.801 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:15.160 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:15.161 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:15.161 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:15.161 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:15.161 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:15.162 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:15.162 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:15.162 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:15.162 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:15.163 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:15.163 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:15.163 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:15.163 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:15.164 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:15.164 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:15.164 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:15.165 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:15.165 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:15.165 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:15.165 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:15.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:15.166 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:15.166 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:15.166 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:15.167 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:15.167 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:15.167 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:15.168 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:15.168 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:15.168 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:15.168 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:15.169 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:15.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:15.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:15.169 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:15.169 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:15.170 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:15.170 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:36:15.170 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:15.170 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:15.171 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:15.171 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:36:15.171 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:15.171 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:15.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:15.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:15.172 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:15.172 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:15.172 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:15.173 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:15.173 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:15.173 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:15.173 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:15.174 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:15.174 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:15.174 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:15.174 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:15.175 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:15.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:15.175 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:15.175 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:15.175 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:15.176 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:15.176 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:15.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:36:20.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:20.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:20.181 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:20.255 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:20.256 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:20.490 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:20.491 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:20.491 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:20.491 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:20.491 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:20.492 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:20.492 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:20.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:20.492 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:20.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:20.493 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:20.493 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:20.493 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:20.493 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:20.494 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:20.494 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:20.494 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:20.494 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:20.495 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:20.495 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:20.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:20.495 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:20.496 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:20.496 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:20.496 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:20.496 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:20.496 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:20.497 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:20.497 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:20.497 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:20.497 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:20.498 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:20.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:20.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:20.498 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:20.499 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:20.499 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:20.499 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:36:20.499 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:20.499 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:20.500 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:20.500 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:36:20.500 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:20.500 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:20.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:20.501 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:20.501 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:20.501 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:20.501 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:20.502 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:20.502 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:20.502 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:20.502 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:20.503 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:20.503 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:20.503 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:20.503 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:20.504 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:20.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:20.504 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:20.504 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:20.505 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:20.505 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:20.505 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:20.506 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.544174_161.1 2021-09-06T11:36:25.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:25.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:25.509 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:25.600 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:25.600 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:25.801 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:25.802 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:25.803 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:25.804 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:25.804 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:25.804 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:25.804 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:25.805 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:25.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:25.807 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:25.808 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:25.809 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:36:25.810 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:36:25.811 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:25.812 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:25.813 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:25.814 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:25.815 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:25.816 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:25.816 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:25.817 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.544174_161.1 2021-09-06T11:36:30.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:30.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:30.820 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:30.914 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:30.914 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:31.176 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:31.176 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:31.177 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:31.177 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:31.177 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:31.177 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:31.178 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:31.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:31.178 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:31.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:31.179 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:31.179 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:31.179 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:31.179 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:31.180 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:31.180 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:31.180 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:31.180 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:31.181 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:31.181 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:31.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:31.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:31.181 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:31.182 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:31.182 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:31.182 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:31.182 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:31.183 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:31.183 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:31.183 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:31.183 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:31.184 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:31.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:31.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:31.184 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:31.185 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:31.185 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:31.185 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:36:31.185 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:31.185 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:31.186 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:31.186 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:36:31.186 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:31.187 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:31.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:31.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:31.187 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:31.188 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:31.188 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:31.188 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:31.188 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:31.189 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:31.189 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:31.189 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:31.189 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:31.190 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:31.190 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:31.190 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:31.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:31.191 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:31.191 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:31.191 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:31.191 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:31.192 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:31.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.484822_152.1 2021-09-06T11:36:36.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:36.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:36.196 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:36.283 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:36.283 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:36.460 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:36.461 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:36.461 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:36.461 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:36.461 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:36.462 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:36.462 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:36.462 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:36.462 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:36.463 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:36.463 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:36.463 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:36.463 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:36.463 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:36.464 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:36.464 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:36.464 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:36.464 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:36.465 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:36.465 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:36.465 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:36.465 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:36.466 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:36.466 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:36.466 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:36.466 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:36.467 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:36.467 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:36.467 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:36.467 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:36.467 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:36.468 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:36.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:36.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:36.468 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:36.469 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:36.469 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:36.469 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:36:36.469 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:36.469 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:36.470 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:36.470 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:36:36.470 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:36.470 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:36.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:36.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:36.471 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:36.471 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:36.472 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:36.472 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:36.472 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:36.472 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:36.472 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:36.473 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:36.473 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:36.473 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:36.474 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:36.474 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:36.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:36.474 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:36.474 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:36.475 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:36.475 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:36.475 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:36.476 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.484822_152.1 2021-09-06T11:36:41.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:41.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:41.479 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:41.586 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:41.586 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:41.918 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:41.919 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:41.919 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:41.919 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:41.920 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:41.920 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:41.920 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:41.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:41.921 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:41.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:41.921 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:41.921 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:41.922 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:41.922 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:41.922 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:41.922 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:41.922 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:41.923 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:41.923 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:41.923 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:41.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:41.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:41.924 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:41.924 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:41.924 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:41.924 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:41.925 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:41.925 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:41.925 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:41.925 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:41.925 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:41.926 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:41.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:41.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:41.926 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:41.926 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:41.927 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:41.927 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:36:41.927 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:41.927 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:41.928 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:41.928 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:36:41.928 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:41.928 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:41.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:41.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:41.929 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:41.929 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:41.929 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:41.929 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:41.930 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:41.930 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:41.930 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:41.930 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:41.931 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:41.931 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:41.931 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:41.931 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:41.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:41.932 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:41.932 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:41.932 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:41.932 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:41.933 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:41.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:36:46.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:46.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:46.937 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:47.036 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:47.037 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:47.195 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:47.195 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:47.196 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:47.197 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:47.198 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:47.199 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:47.200 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:47.201 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:47.202 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:36:47.203 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:36:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:47.204 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:47.205 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:47.206 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:47.207 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:47.208 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:47.209 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:47.210 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:36:52.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:52.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:52.213 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:52.302 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:52.302 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:53.060 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:53.060 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:53.060 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:53.061 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:53.061 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:53.061 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:53.061 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:53.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:53.062 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:53.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:53.062 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:53.063 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:53.063 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:53.063 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:53.063 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:53.063 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:53.064 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:53.064 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:53.064 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:53.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:53.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:53.065 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:53.065 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:53.065 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:53.066 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:53.066 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:53.066 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:53.066 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:53.066 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:53.067 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:53.067 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:53.067 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:53.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:53.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:53.068 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:53.068 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:53.068 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:53.068 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:36:53.069 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:53.069 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:53.069 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:53.069 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:36:53.070 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:53.070 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:53.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:53.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:53.070 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:53.071 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:53.071 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:53.071 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:53.071 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:53.072 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:53.072 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:53.072 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:53.072 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:53.072 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:53.073 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:53.073 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:53.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:53.073 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:53.074 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:53.074 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:53.074 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:53.074 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:53.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.508454_157.1 2021-09-06T11:36:58.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:58.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:36:58.079 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:36:58.164 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:36:58.165 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:36:58.817 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:36:58.818 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:36:58.818 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:36:58.818 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:36:58.818 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:36:58.819 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:36:58.819 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:58.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:58.819 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:36:58.820 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:58.820 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:36:58.820 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:58.820 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:58.821 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:36:58.821 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:58.821 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:36:58.821 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:58.822 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:36:58.822 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:58.822 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:58.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:58.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:58.823 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:36:58.823 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:58.823 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:58.824 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:36:58.824 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:58.824 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:58.824 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:58.825 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:36:58.825 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:58.825 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:58.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:58.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:58.826 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:36:58.826 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:58.826 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:58.826 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:36:58.827 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:58.827 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:58.827 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:58.827 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:36:58.828 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:58.828 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:58.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:58.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:36:58.829 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:36:58.829 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:36:58.829 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:36:58.829 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:36:58.829 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:36:58.830 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:36:58.830 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:36:58.830 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:36:58.830 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:36:58.831 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:58.831 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:58.831 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:36:58.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:36:58.831 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:36:58.832 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:36:58.832 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:36:58.832 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:36:58.832 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:36:58.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:37:03.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:03.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:03.836 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:03.914 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:03.914 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:04.415 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:04.416 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:04.417 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:04.418 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:04.419 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:04.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:04.421 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:04.422 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:04.423 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.611335_165.1", 2021-09-06T11:37:04.424 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.611335Z", 2021-09-06T11:37:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:04.425 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:04.426 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:04.427 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:04.428 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:04.429 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:04.430 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:04.430 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:04.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.611335_165.1 2021-09-06T11:37:09.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:09.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:09.433 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:09.521 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:09.521 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:10.004 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:10.005 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:10.005 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:10.005 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:10.006 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:10.006 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:10.006 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:10.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:10.007 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:10.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:10.007 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:10.007 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:10.008 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:10.008 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:10.008 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:10.008 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:10.009 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:10.009 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:10.009 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:10.009 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:10.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:10.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:10.010 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:10.010 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:10.011 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:10.011 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:10.011 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:10.011 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:10.011 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:10.012 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:10.012 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:10.012 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:10.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:10.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:10.013 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:10.013 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:10.013 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:10.014 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:37:10.014 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:10.014 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:10.014 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:10.014 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:37:10.015 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:10.015 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:10.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:10.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:10.016 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:10.016 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:10.016 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:10.016 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:10.017 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:10.017 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:10.017 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:10.017 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:10.018 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:10.018 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:10.018 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:10.018 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:10.018 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:10.019 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:10.019 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:10.019 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:10.019 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:10.020 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:10.021 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.503662_155.1 2021-09-06T11:37:15.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:15.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:15.024 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:15.116 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:15.117 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:15.698 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:15.699 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:15.699 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:15.699 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:15.700 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:15.700 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:15.700 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:15.700 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:15.701 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:15.701 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:15.701 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:15.701 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:15.702 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:15.702 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:15.702 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:15.702 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:15.702 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:15.703 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:15.703 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:15.703 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:15.703 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:15.704 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:15.704 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:15.704 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:15.704 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:15.705 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:15.705 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:15.705 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:15.705 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:15.706 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:15.706 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:15.706 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:15.706 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:15.706 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:15.707 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:15.707 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:15.707 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:15.707 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:37:15.708 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:15.708 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:15.708 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:15.708 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:37:15.709 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:15.709 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:15.709 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:15.709 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:15.709 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:15.710 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:15.710 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:15.710 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:15.710 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:15.711 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:15.711 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:15.711 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:15.711 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:15.712 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:15.712 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:15.712 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:15.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:15.713 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:15.713 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:15.713 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:15.713 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:15.714 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:15.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:37:20.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:20.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:20.718 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:20.807 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:20.808 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:21.420 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:21.421 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:21.421 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:21.421 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:21.421 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:21.422 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:21.422 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:21.422 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:21.422 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:21.423 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:21.423 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:21.423 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:21.423 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:21.423 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:21.424 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:21.424 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:21.424 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:21.424 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:21.425 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:21.425 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:21.425 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:21.425 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:21.426 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:21.426 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:21.426 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:21.426 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:21.427 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:21.427 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:21.427 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:21.427 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:21.427 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:21.428 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:21.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:21.428 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:21.428 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:21.429 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:21.429 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:21.429 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:37:21.429 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:21.430 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:21.430 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:21.430 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:37:21.430 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:21.431 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:21.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:21.431 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:21.431 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:21.431 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:21.432 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:21.432 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:21.432 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:21.432 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:21.433 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:21.433 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:21.433 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:21.433 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:21.434 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:21.434 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:21.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:21.434 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:21.434 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:21.435 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:21.435 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:21.435 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:21.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:37:26.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:26.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:26.440 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:26.490 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:26.490 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:27.247 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:27.248 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:27.248 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:27.248 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:27.249 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:27.249 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:27.250 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:27.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:27.251 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:27.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:27.252 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:27.252 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:27.252 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:27.253 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:27.253 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:27.254 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:27.254 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:27.254 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:27.255 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:27.255 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:27.256 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:27.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:27.257 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:27.257 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:27.257 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:27.258 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:27.258 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:27.259 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:27.259 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:27.260 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:27.260 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:27.260 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:27.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:27.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:27.262 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:27.262 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:27.263 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:27.263 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:37:27.264 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:27.264 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:27.264 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:27.265 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:37:27.265 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:27.266 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:27.266 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:27.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:27.267 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:27.267 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:27.268 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:27.268 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:27.269 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:27.269 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:27.270 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:27.270 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:27.271 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:27.271 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:27.271 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:27.272 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:27.272 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:27.273 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:27.273 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:27.274 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:27.274 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:27.274 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:27.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.544174_161.1 2021-09-06T11:37:32.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:32.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:32.278 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:32.358 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:32.358 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:32.559 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:32.559 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:32.560 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:32.560 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:32.560 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:32.560 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:32.561 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:32.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:32.561 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:32.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:32.562 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:32.562 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:32.563 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:32.563 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:32.563 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:32.563 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:32.563 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:32.564 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:32.564 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:32.564 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:32.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:32.565 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:32.565 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:32.565 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:32.565 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:32.565 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:32.566 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:32.566 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:32.566 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:32.566 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:32.567 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:32.567 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:32.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:32.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:32.567 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:32.568 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:32.568 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:32.568 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:37:32.568 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:32.569 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:32.569 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:32.569 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:37:32.569 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:32.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:32.570 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:32.570 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:32.570 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:32.570 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:32.571 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:32.571 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:32.571 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:32.571 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:32.572 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:32.572 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:32.572 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:32.572 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:32.572 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:32.573 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:32.573 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:32.573 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:32.573 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:32.574 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:32.574 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:32.574 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:32.576 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.507089_156.1 2021-09-06T11:37:37.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:37.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:37.578 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:37.671 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:37.672 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:37.910 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:37.911 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:37.911 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:37.911 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:37.911 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:37.912 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:37.912 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:37.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:37.912 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:37.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:37.913 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:37.913 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:37.913 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:37.914 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:37.914 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:37.914 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:37.914 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:37.915 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:37.915 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:37.915 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:37.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:37.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:37.916 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:37.916 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:37.917 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:37.917 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:37.917 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:37.917 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:37.918 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:37.918 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:37.918 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:37.918 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:37.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:37.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:37.919 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:37.919 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:37.920 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:37.920 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:37:37.920 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:37.920 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:37.921 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:37.921 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:37:37.921 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:37.921 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:37.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:37.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:37.922 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:37.922 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:37.923 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:37.923 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:37.923 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:37.923 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:37.924 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:37.924 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:37.924 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:37.924 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:37.925 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:37.925 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:37.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:37.925 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:37.926 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:37.926 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:37.926 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:37.926 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:37.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:37:42.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:42.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:42.930 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:42.978 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:42.978 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:43.050 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:43.050 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:43.050 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:43.050 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:43.051 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:43.051 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:43.051 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:43.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:43.052 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:43.052 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:43.052 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:43.052 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:43.052 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:43.053 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:43.053 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:43.053 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:43.053 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:43.054 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:43.054 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:43.054 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:43.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:43.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:43.055 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:43.055 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:43.055 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:43.056 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:43.056 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:43.056 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:43.056 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:43.057 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:43.057 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:43.057 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:43.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:43.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:43.058 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:43.058 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:43.058 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:43.059 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:37:43.059 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:43.059 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:43.059 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:43.060 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:37:43.060 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:43.060 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:43.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:43.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:43.061 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:43.061 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:43.061 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:43.062 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:43.062 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:43.062 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:43.062 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:43.063 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:43.063 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:43.063 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:43.063 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:43.063 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:43.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:43.064 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:43.064 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:43.064 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:43.065 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:43.065 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:43.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:37:48.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:48.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:48.069 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:48.153 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:48.154 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:48.646 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:48.646 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:48.646 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:48.647 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:48.647 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:48.647 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:48.647 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:48.648 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:48.648 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:48.648 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:48.648 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:48.648 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:48.649 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:48.649 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:48.649 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:48.649 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:48.650 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:48.650 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:48.650 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:48.650 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:48.651 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:48.651 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:48.651 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:48.651 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:48.651 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:48.652 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:48.652 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:48.652 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:48.652 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:48.653 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:48.653 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:48.653 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:48.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:48.653 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:48.654 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:48.654 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:48.654 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:48.654 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:37:48.655 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:48.655 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:48.655 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:48.655 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:37:48.656 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:48.656 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:48.656 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:48.656 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:48.657 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:48.657 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:48.657 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:48.657 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:48.657 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:48.658 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:48.658 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:48.658 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:48.658 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:48.659 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:48.659 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:48.659 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:48.659 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:48.659 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:48.660 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:48.660 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:48.660 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:48.660 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:48.662 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:37:53.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:53.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:53.665 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:53.740 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:53.740 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:54.000 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:54.001 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:54.002 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:54.003 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:54.004 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:54.005 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:54.006 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:54.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:54.008 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:54.009 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:54.010 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:37:54.011 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:37:54.012 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:54.014 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:54.015 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:54.016 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:54.017 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:54.017 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:54.019 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:37:59.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:59.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:37:59.023 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:37:59.069 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:37:59.070 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:37:59.142 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:37:59.143 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:37:59.143 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:37:59.144 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:37:59.144 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:37:59.144 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:37:59.145 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:59.145 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:59.145 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:37:59.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:59.146 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:37:59.147 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:59.147 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:59.147 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:37:59.148 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:59.148 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:37:59.148 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:59.149 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:37:59.149 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:59.149 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:59.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:59.150 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:59.150 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:37:59.150 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:59.151 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:59.151 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:37:59.151 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:59.151 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:59.151 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:59.152 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:37:59.152 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:59.152 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:59.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:59.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:59.153 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:37:59.153 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:59.153 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:59.153 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:37:59.154 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:59.154 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:59.154 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:59.154 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:37:59.155 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:59.155 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:59.155 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:59.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:37:59.156 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:37:59.156 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:37:59.156 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:37:59.156 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:37:59.157 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:37:59.157 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:37:59.157 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:37:59.157 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:37:59.157 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:37:59.158 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:59.158 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:59.158 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:37:59.158 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:37:59.159 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:37:59.159 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:37:59.159 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:37:59.160 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:37:59.160 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:37:59.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.544174_161.1 2021-09-06T11:38:04.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:04.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:04.164 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:04.226 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:04.227 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:04.327 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:04.327 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:04.335 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:04.335 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:04.335 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:04.336 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:04.336 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:04.336 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:04.337 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:38:04.337 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:04.337 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:04.337 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:38:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:04.343 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:04.344 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.738114_171.1 2021-09-06T11:38:09.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:09.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:09.346 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:09.430 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:09.431 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:09.581 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:09.582 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:09.582 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:09.582 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:09.582 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:09.583 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:09.583 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:09.583 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:09.583 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:09.584 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:09.584 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:09.584 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:09.584 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:09.584 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:09.585 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:09.585 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:09.586 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:09.586 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:09.586 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:09.587 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:09.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:09.588 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:09.588 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:09.589 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:09.589 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:09.589 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:09.589 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:09.590 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:09.590 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:09.590 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:09.590 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:09.591 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:09.591 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:09.591 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:09.591 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:09.592 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:09.592 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:09.592 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:38:09.592 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:09.592 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:09.593 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:09.593 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:38:09.593 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:09.593 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:09.594 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:09.594 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:09.594 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:09.594 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:09.595 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:09.595 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:09.595 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:09.595 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:09.596 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:09.596 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:09.597 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:09.597 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:09.598 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:09.598 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:09.598 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:09.598 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:09.599 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:09.599 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:09.599 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:09.600 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:09.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.484822_152.1 2021-09-06T11:38:14.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:14.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:14.604 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:14.670 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:14.670 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:14.745 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:14.745 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:14.746 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:14.746 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:14.746 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:14.747 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:14.747 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:14.747 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:14.747 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:14.747 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:14.748 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:14.748 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:14.748 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:14.748 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:14.749 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:14.749 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:14.749 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:14.749 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:14.749 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:14.750 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:14.750 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:14.750 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:14.750 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:14.751 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:14.751 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:14.751 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:14.751 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:14.752 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:14.752 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:14.752 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:14.752 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:14.752 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:14.753 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:14.753 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:14.753 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:14.753 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:14.753 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:14.754 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:38:14.754 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:14.754 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:14.754 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:14.755 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:38:14.755 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:14.755 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:14.755 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:14.756 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:14.756 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:14.756 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:14.756 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:14.756 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:14.757 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:14.757 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:14.757 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:14.757 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:14.758 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:14.758 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:14.758 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:14.758 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:14.759 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:14.759 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:14.759 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:14.759 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:14.760 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:14.760 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:14.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.508454_157.1 2021-09-06T11:38:19.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:19.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:19.764 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:19.820 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:19.820 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:19.905 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:19.906 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:19.906 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:19.907 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:19.907 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:19.907 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:19.907 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:19.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:19.908 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:19.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:19.908 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:19.909 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:19.909 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:19.909 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:19.909 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:19.910 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:19.910 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:19.910 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:19.910 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:19.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:19.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:19.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:19.911 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:19.911 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:19.912 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:19.912 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:19.912 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:19.912 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:19.913 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:19.913 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:19.913 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:19.913 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:19.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:19.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:19.914 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:19.914 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:19.914 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:19.915 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:38:19.915 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:19.915 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:19.915 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:19.915 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:38:19.916 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:19.916 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:19.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:19.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:19.917 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:19.917 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:19.917 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:19.917 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:19.918 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:19.918 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:19.918 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:19.918 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:19.919 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:19.919 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:19.919 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:19.919 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:19.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:19.920 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:19.920 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:19.920 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:19.920 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:19.921 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:19.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.484822_152.1 2021-09-06T11:38:24.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:24.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:24.925 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:24.980 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:24.981 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:25.043 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:25.043 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:25.043 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:25.044 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:25.044 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:25.044 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:25.044 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:25.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:25.045 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:25.045 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:25.045 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:25.046 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:25.046 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:25.046 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:25.046 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:25.047 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:25.047 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:25.047 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:25.047 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:25.047 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:25.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:25.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:25.048 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:25.049 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:25.049 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:25.049 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:25.049 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:25.049 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:25.050 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:25.050 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:25.050 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:25.050 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:25.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:25.051 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:25.051 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:25.051 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:25.052 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:25.052 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:38:25.052 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:25.052 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:25.053 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:25.053 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:38:25.053 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:25.053 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:25.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:25.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:25.054 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:25.054 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:25.055 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:25.055 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:25.055 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:25.055 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:25.055 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:25.056 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:25.056 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:25.056 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:25.056 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:25.057 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:25.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:25.057 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:25.057 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:25.057 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:25.058 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:25.058 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:25.059 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.503662_155.1 2021-09-06T11:38:30.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:30.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:30.062 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:30.116 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:30.116 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:30.183 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:30.183 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:30.184 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:30.184 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:30.185 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:30.185 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:30.185 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:30.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:30.185 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:30.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:30.186 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:30.186 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:30.186 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:30.187 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:30.187 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:30.187 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:30.187 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:30.188 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:30.188 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:30.188 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:30.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:30.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:30.189 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:30.189 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:30.189 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:30.189 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:30.190 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:30.190 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:30.190 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:30.190 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:30.190 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:30.195 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:38:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:38:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:30.202 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:30.202 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:30.204 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:30.204 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:30.204 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:30.205 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.508454_157.1 2021-09-06T11:38:35.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:35.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:35.209 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:35.279 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:35.280 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:35.419 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:35.419 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:35.420 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:35.420 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:35.420 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:35.421 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:35.421 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:35.421 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:35.422 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:35.422 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:35.422 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:35.423 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:35.423 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:35.424 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:35.424 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:35.424 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:35.424 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:35.425 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:35.425 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:35.425 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:35.425 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:35.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:35.426 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:35.426 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:35.426 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:35.427 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:35.427 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:35.427 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:35.427 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:35.428 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:35.428 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:35.428 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:35.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:35.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:35.429 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:35.429 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:35.430 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:35.430 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:38:35.430 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:35.430 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:35.431 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:35.431 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:38:35.431 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:35.431 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:35.432 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:35.432 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:35.432 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:35.432 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:35.433 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:35.433 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:35.433 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:35.434 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:35.434 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:35.435 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:35.435 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:35.435 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:35.435 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:35.436 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:35.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:35.436 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:35.437 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:35.437 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:35.437 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:35.437 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:35.439 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.503662_155.1 2021-09-06T11:38:40.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:40.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:40.442 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:40.528 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:40.528 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:40.717 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:40.717 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:40.718 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:40.718 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:40.718 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:40.718 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:40.719 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:40.719 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:40.719 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:40.719 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:40.720 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:40.720 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:40.720 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:40.720 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:40.720 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:40.721 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:40.721 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:40.721 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:40.721 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:40.722 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:40.722 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:40.722 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:40.722 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:40.722 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:40.723 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:40.723 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:40.723 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:40.723 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:40.724 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:40.724 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:40.724 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:40.724 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:40.725 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:40.725 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:40.725 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:40.725 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:40.725 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:40.726 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:38:40.726 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:40.726 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:40.726 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:40.727 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:38:40.727 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:40.727 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:40.727 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:40.728 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:40.728 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:40.728 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:40.729 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:40.729 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:40.729 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:40.729 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:40.730 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:40.730 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:40.731 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:40.731 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:40.731 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:40.731 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:40.732 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:40.732 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:40.732 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:40.732 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:40.733 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:40.733 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:40.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.738114_171.1 2021-09-06T11:38:45.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:45.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:45.737 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:45.781 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:45.781 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:45.843 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:45.843 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:45.843 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:45.844 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:45.844 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:45.844 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:45.844 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:45.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:45.845 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:45.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:45.845 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:45.845 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:45.846 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:45.846 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:45.846 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:45.846 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:45.847 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:45.847 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:45.847 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:45.847 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:45.847 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:45.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:45.848 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:45.848 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:45.848 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:45.849 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:45.849 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:45.849 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:45.849 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:45.850 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:45.850 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:45.850 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:45.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:45.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:45.851 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:45.851 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:45.851 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:45.851 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:38:45.852 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:45.852 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:45.852 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:45.852 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:38:45.853 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:45.853 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:45.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:45.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:45.853 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:45.854 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:45.854 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:45.854 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:45.854 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:45.855 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:45.855 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:45.855 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:45.855 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:45.856 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:45.856 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:45.856 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:45.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:45.856 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:45.857 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:45.857 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:45.857 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:45.857 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:45.859 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:38:50.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:50.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:50.861 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:50.903 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:50.904 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:50.977 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:50.978 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:50.978 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:50.978 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:50.979 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:50.979 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:50.979 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:50.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:50.980 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:50.980 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:50.980 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:50.981 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:50.981 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:50.981 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:50.981 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:50.982 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:50.982 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:50.982 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:50.982 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:50.983 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:50.983 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:50.983 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:50.983 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:50.984 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:50.984 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:50.984 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:50.984 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:50.985 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:50.985 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:50.985 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:50.985 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:50.985 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:50.986 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:50.986 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:50.986 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:50.986 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:50.987 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:50.987 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:38:50.987 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:50.987 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:50.988 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:50.988 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:38:50.988 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:50.988 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:50.988 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:50.989 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:50.989 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:50.989 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:50.989 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:50.990 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:50.990 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:50.990 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:50.991 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:50.991 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:50.991 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:50.991 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:50.991 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:50.992 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:50.992 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:50.992 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:50.992 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:50.993 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:50.993 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:50.993 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:50.995 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.503662_155.1 2021-09-06T11:38:55.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:55.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:38:55.997 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:38:56.040 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:38:56.040 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:38:56.107 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:38:56.107 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:38:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:38:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:38:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:38:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:38:56.108 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:56.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:56.109 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:38:56.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:56.109 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:38:56.110 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:56.110 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:56.110 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:38:56.110 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:56.110 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:38:56.111 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:56.111 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:38:56.111 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:56.111 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:56.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:56.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:56.112 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:38:56.112 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:56.113 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:56.113 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:38:56.113 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:56.113 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:56.113 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:56.114 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:38:56.114 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:56.114 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:56.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:56.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:56.115 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:38:56.115 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:56.115 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:56.116 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:38:56.116 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:56.116 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:56.116 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:56.117 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:38:56.117 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:56.117 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:56.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:56.118 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:38:56.118 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:38:56.118 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:38:56.118 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:38:56.118 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:38:56.119 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:38:56.119 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:38:56.119 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:38:56.119 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:38:56.120 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:38:56.120 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:56.120 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:56.120 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:38:56.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:38:56.121 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:38:56.121 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:38:56.121 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:38:56.121 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:38:56.122 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:38:56.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:39:01.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:01.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:01.125 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:01.175 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:01.175 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:01.255 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:01.255 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:01.255 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:01.256 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:01.256 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:01.256 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:01.256 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:01.256 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:01.257 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:01.257 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:01.257 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:01.257 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:01.257 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:01.258 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:01.258 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:01.258 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:01.258 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:01.259 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:01.259 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:01.259 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:01.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:01.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:01.260 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:01.260 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:01.260 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:01.261 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:01.261 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:01.261 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:01.261 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:01.262 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:01.262 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:01.262 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:01.262 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:01.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:01.263 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:01.263 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:01.264 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:01.264 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:39:01.264 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:01.264 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:01.265 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:01.265 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:39:01.265 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:01.266 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:01.266 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:01.266 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:01.267 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:01.267 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:01.267 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:01.267 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:01.268 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:01.268 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:01.268 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:01.268 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:01.268 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:01.269 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:01.269 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:01.269 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:01.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:01.269 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:01.270 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:01.270 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:01.270 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:01.271 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:01.272 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:39:06.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:06.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:06.275 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:06.318 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:06.318 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:06.382 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:06.382 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:06.382 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:06.382 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:06.383 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:06.383 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:06.383 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:06.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:06.384 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:06.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:06.384 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:06.384 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:06.385 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:06.385 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:06.385 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:06.385 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:06.386 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:06.386 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:06.386 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:06.386 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:06.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:06.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:06.387 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:06.387 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:06.387 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:06.387 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:06.388 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:06.388 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:06.388 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:06.388 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:06.389 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:06.389 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:06.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:06.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:06.389 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:06.390 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:06.390 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:06.390 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:39:06.390 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:06.391 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:06.391 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:06.391 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:39:06.391 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:06.391 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:06.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:06.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:06.392 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:06.392 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:06.392 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:06.393 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:06.393 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:06.393 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:06.393 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:06.394 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:06.394 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:06.394 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:06.394 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:06.394 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:06.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:06.395 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:06.395 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:06.395 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:06.396 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:06.396 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:06.397 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.544174_161.1 2021-09-06T11:39:11.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:11.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:11.400 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:11.476 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:11.476 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:11.556 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:11.556 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:11.556 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:11.557 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:11.557 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:11.557 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:11.557 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:11.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:11.558 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:11.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:11.558 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:11.558 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:11.559 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:11.559 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:11.559 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:11.559 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:11.560 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:11.560 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:11.560 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:11.560 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:11.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:11.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:11.561 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:11.561 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:11.561 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:11.562 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:11.562 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:11.562 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:11.562 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:11.563 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:11.563 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:11.563 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:11.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:11.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:11.564 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:11.564 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:11.564 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:11.564 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.763245_176.1", 2021-09-06T11:39:11.565 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:11.565 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:11.565 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:11.565 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.763245Z", 2021-09-06T11:39:11.565 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:11.566 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:11.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:11.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:11.566 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:11.567 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:11.567 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:11.567 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:11.567 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:11.568 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:11.568 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:11.568 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:11.568 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:11.568 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:11.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:11.569 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:11.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:11.569 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:11.570 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:11.570 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:11.570 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:11.570 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:11.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.763245_176.1 2021-09-06T11:39:16.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:16.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:16.574 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:16.832 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:16.833 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:16.899 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:16.899 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:16.900 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:16.900 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:16.900 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:16.900 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:16.901 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:16.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:16.901 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:16.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:16.901 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:16.902 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:16.902 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:16.902 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:16.902 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:16.903 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:16.903 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:16.903 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:16.903 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:16.903 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:16.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:16.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:16.904 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:16.904 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:16.904 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:16.905 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:16.905 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:16.905 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:16.905 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:16.906 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:16.906 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:16.906 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:16.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:16.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:16.907 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:16.907 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:16.907 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:16.907 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.759827_175.1", 2021-09-06T11:39:16.908 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:16.908 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:16.908 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:16.908 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.759827Z", 2021-09-06T11:39:16.908 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:16.909 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:16.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:16.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:16.909 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:16.910 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:16.910 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:16.910 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:16.910 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:16.910 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:16.911 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:16.911 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:16.911 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:16.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:16.912 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:16.912 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:16.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:16.912 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:16.913 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:16.913 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:16.913 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:16.913 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:16.914 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.759827_175.1 2021-09-06T11:39:21.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:21.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:21.917 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:22.987 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:22.987 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:23.059 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:23.060 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:23.060 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:23.060 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:23.060 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:23.061 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:23.061 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:23.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:23.061 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:23.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:23.062 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:23.062 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:23.062 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:23.062 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:23.063 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:23.063 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:23.063 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:23.063 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:23.064 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:23.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:23.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:23.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:23.065 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:23.065 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:23.065 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:23.065 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:23.065 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:23.066 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:23.066 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:23.066 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:23.066 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:23.067 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:23.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:23.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:23.067 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:23.067 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:23.068 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:23.068 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:39:23.068 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:23.068 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:23.069 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:23.069 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:39:23.069 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:23.069 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:23.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:23.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:23.070 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:23.070 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:23.070 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:23.071 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:23.071 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:23.071 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:23.071 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:23.071 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:23.072 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:23.072 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:23.072 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:23.073 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:23.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:23.073 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:23.073 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:23.073 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:23.074 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:23.074 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:23.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.507089_156.1 2021-09-06T11:39:28.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:28.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:28.078 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:28.213 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:28.214 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:28.291 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:28.291 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:28.292 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:28.292 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:28.292 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:28.292 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:28.293 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:28.293 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:28.293 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:28.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:28.294 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:28.294 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:28.294 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:28.294 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:28.295 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:28.295 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:28.295 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:28.295 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:28.295 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:28.296 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:28.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:28.296 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:28.297 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:28.297 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:28.297 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:28.297 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:28.298 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:28.298 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:28.298 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:28.298 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:28.299 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:28.299 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:28.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:28.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:28.299 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:28.300 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:28.300 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:28.300 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:39:28.300 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:28.301 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:28.301 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:28.301 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:39:28.301 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:28.302 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:28.302 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:28.302 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:28.302 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:28.303 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:28.303 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:28.303 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:28.304 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:28.304 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:28.304 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:28.304 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:28.304 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:28.305 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:28.305 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:28.305 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:28.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:28.306 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:28.306 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:28.306 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:28.306 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:28.307 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:28.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.507089_156.1 2021-09-06T11:39:33.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:33.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:33.311 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:33.753 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:33.753 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:34.239 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:34.240 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:34.240 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:34.240 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:34.241 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:34.241 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:34.241 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:34.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:34.242 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:34.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:34.242 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:34.242 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:34.242 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:34.243 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:34.243 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:34.243 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:34.243 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:34.244 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:34.244 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:34.244 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:34.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:34.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:34.245 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:34.245 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:34.245 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:34.245 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:34.246 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:34.246 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:34.246 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:34.246 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:34.247 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:34.247 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:34.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:34.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:34.247 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:34.248 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:34.248 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:34.248 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:39:34.249 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:34.249 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:34.249 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:34.249 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:39:34.250 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:34.250 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:34.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:34.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:34.250 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:34.251 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:34.251 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:34.251 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:34.251 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:34.252 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:34.252 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:34.252 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:34.252 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:34.252 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:34.253 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:34.253 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:34.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:34.253 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:34.254 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:34.254 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:34.254 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:34.254 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:34.255 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.508454_157.1 2021-09-06T11:39:39.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:39.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:39.258 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:39.504 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:39.504 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:39.777 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:39.778 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:39.778 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:39.778 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:39.779 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:39.779 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:39.779 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:39.779 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:39.780 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:39.780 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:39.780 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:39.780 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:39.781 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:39.781 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:39.781 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:39.781 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:39.782 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:39.782 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:39.782 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:39.782 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:39.783 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:39.783 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:39.783 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:39.783 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:39.784 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:39.784 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:39.784 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:39.784 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:39.785 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:39.785 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:39.785 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:39.785 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:39.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:39.786 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:39.786 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:39.786 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:39.787 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:39.787 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:39:39.787 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:39.787 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:39.787 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:39.788 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:39:39.788 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:39.788 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:39.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:39.789 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:39.789 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:39.789 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:39.789 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:39.789 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:39.790 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:39.790 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:39.790 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:39.790 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:39.791 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:39.791 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:39.791 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:39.791 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:39.792 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:39.792 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:39.792 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:39.792 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:39.792 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:39.793 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:39.794 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:39:44.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:44.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:44.796 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:44.849 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:44.850 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:44.925 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:44.932 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:44.932 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:44.932 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:44.932 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:44.934 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:44.934 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:44.934 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:44.934 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:39:44.935 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:44.935 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:44.935 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:44.935 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:39:44.935 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:44.936 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:44.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:44.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:44.936 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:44.937 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:44.937 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:44.937 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:44.937 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:44.938 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:44.938 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:44.938 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:44.938 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:44.940 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:44.940 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:44.940 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:44.940 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:44.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.508454_157.1 2021-09-06T11:39:49.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:49.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:49.944 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:50.003 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:50.004 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:50.099 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:50.099 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:50.100 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:50.100 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:50.100 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:50.100 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:50.101 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:50.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:50.101 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:50.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:50.102 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:50.102 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:50.102 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:50.102 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:50.103 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:50.103 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:50.103 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:50.104 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:50.104 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:50.104 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:50.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:50.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:50.105 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:50.105 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:50.106 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:50.106 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:50.106 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:50.106 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:50.107 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:50.107 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:50.107 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:50.107 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:50.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:50.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:50.108 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:50.108 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:50.109 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:50.109 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.543029_160.1", 2021-09-06T11:39:50.109 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:50.109 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:50.110 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:50.110 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.543029Z", 2021-09-06T11:39:50.110 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:50.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:50.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:50.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:50.111 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:50.111 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:50.111 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:50.112 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:50.112 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:50.112 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:50.112 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:50.113 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:50.113 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:50.113 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:50.113 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:50.113 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:50.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:50.114 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:50.114 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:50.115 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:50.115 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:50.115 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:50.116 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.543029_160.1 2021-09-06T11:39:55.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:55.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:39:55.119 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:39:55.196 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:39:55.196 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:39:55.292 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:39:55.292 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:39:55.292 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:39:55.293 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:39:55.293 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:39:55.293 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:39:55.293 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:55.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:55.294 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:39:55.294 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:55.294 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:39:55.294 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:55.295 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:55.295 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:39:55.295 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:55.295 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:39:55.296 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:55.296 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:39:55.296 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:55.296 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:55.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:55.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:55.297 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:39:55.297 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:55.298 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:55.298 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:39:55.298 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:55.298 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:55.298 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:55.299 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:39:55.299 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:55.299 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:55.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:55.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:55.300 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:39:55.300 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:55.300 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:55.300 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T11:39:55.301 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:55.301 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:55.301 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:55.301 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T11:39:55.302 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:55.302 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:55.302 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:55.302 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:39:55.302 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:39:55.303 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:39:55.303 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:39:55.303 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:39:55.303 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:39:55.303 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:39:55.304 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:39:55.304 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:39:55.304 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:39:55.304 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:55.305 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:55.305 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:39:55.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:39:55.305 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:39:55.306 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:39:55.306 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:39:55.306 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:39:55.306 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:39:55.307 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.489232_154.1 2021-09-06T11:40:00.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:00.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:00.310 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:00.358 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:00.358 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:00.427 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:00.427 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:00.427 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:00.427 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:00.428 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:00.428 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:00.428 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:00.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:00.429 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:00.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:00.429 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:00.429 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:00.430 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:00.430 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:00.430 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:00.430 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:00.430 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:00.431 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:00.431 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:00.431 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:00.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:00.431 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:00.432 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:00.432 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:00.432 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:00.432 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:00.432 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:00.433 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:00.433 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:00.433 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:00.433 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:00.433 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:00.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:00.434 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:00.434 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:00.434 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:00.434 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:00.435 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:40:00.435 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:00.435 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:00.435 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:00.435 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:40:00.436 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:00.436 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:00.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:00.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:00.437 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:00.437 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:00.437 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:00.437 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:00.437 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:00.438 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:00.438 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:00.438 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:00.438 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:00.439 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:00.439 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:00.439 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:00.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:00.439 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:00.440 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:00.440 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:00.440 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:00.440 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:00.441 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.544174_161.1 2021-09-06T11:40:05.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:05.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:05.445 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:05.504 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:05.505 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:05.597 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:05.597 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:05.597 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:05.598 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:05.598 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:05.598 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:05.598 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:05.599 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:05.599 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:05.599 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:05.599 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:05.599 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:05.600 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:05.600 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:05.600 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:05.600 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:05.600 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:05.601 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:05.601 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:05.601 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:05.601 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:05.602 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:05.602 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:05.602 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:05.602 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:05.603 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:05.603 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:05.603 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:05.603 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:05.603 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:05.604 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:05.604 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:05.604 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:05.604 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:05.605 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:05.605 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:05.605 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:05.605 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:40:05.605 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:05.606 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:05.606 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:05.606 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:40:05.606 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:05.607 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:05.607 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:05.607 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:05.607 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:05.607 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:05.608 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:05.608 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:05.608 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:05.608 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:05.609 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:05.609 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:05.609 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:05.609 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:05.609 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:05.610 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:05.610 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:05.610 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:05.610 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:05.611 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:05.611 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:05.611 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:05.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:40:10.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:10.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:10.614 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:10.663 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:10.664 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:10.732 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:10.733 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:10.733 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:10.733 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:10.733 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:10.734 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:10.734 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:10.734 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:10.734 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:10.735 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:10.735 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:10.735 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:10.735 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:10.735 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:10.736 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:10.736 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:10.736 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:10.736 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:10.737 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:10.737 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:10.737 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:10.737 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:10.737 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:10.738 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:10.738 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:10.738 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:10.738 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:10.739 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:10.739 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:10.739 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:10.739 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:10.739 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:10.740 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:10.740 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:10.740 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:10.740 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:10.741 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:10.741 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:40:10.741 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:10.741 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:10.741 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:10.742 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:40:10.742 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:10.742 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:10.742 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:10.743 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:10.744 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:10.744 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:10.744 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:10.744 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:10.745 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:10.745 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:10.745 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:10.745 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:10.745 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:10.746 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:10.746 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:10.746 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:10.746 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:10.747 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:10.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928094.616311_216.1 target=1_1630928091.486035_153.1 2021-09-06T11:40:15.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2021-09-06T11:40:15.751 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-39 2021-09-06T11:40:15.786 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-40 2021-09-06T11:40:15.897 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-41 2021-09-06T11:40:15.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:15.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:15.943 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:16.145 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:16.146 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:16.195 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:40:16.211 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:40:16.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:40:16.211 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:16.211 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:40:16.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:40:16.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:40:16.212 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:40:16.212 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:40:16.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:40:16.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:40:16.213 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928415.920208_237.1", 2021-09-06T11:40:16.213 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:40:15.920208Z" 2021-09-06T11:40:16.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:40:16.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:40:16.214 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:16.214 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:40:16.214 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:40:16.215 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:40:16.216 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:40:16.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:16.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:16.217 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:16.292 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:16.292 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:16.561 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:16.562 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:16.562 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:16.562 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:16.563 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:16.563 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:16.563 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:16.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:16.564 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:16.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:16.564 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:16.564 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:16.565 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:16.565 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:16.565 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:16.565 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:16.565 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:16.566 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:16.566 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:16.566 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:16.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:16.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:16.567 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:16.567 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:16.567 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:16.568 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:16.568 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:16.568 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:16.568 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:16.568 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:16.569 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:16.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:16.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:16.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:16.569 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:16.570 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:16.570 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:16.570 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:40:16.570 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:16.571 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:16.571 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:16.571 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:40:16.571 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:16.571 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:16.572 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:16.572 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:16.572 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:16.572 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:16.573 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:16.573 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:16.573 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:16.573 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:16.573 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:16.574 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:16.574 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:16.574 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:16.574 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:16.575 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:16.575 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:16.575 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:16.575 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:16.575 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:16.576 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:16.576 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:16.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:40:21.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:21.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:21.580 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:21.655 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:21.655 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:21.749 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:21.749 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:21.750 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:21.750 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:21.750 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:21.750 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:21.751 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:21.751 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:21.751 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:21.751 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:21.751 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:21.752 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:21.752 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:21.752 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:21.752 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:21.753 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:21.753 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:21.753 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:21.754 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:21.754 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:21.754 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:21.754 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:21.755 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:21.755 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:21.755 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:21.755 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:21.755 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:21.756 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:21.756 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:21.756 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:21.756 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:21.757 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:21.757 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:21.757 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:21.757 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:21.757 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:21.758 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:21.758 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:40:21.758 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:21.758 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:21.758 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:21.759 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:40:21.759 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:21.759 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:21.759 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:21.760 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:21.760 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:21.760 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:21.760 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:21.760 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:21.761 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:21.761 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:21.761 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:21.761 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:21.762 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:21.762 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:21.762 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:21.762 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:21.762 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:21.763 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:21.763 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:21.763 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:21.763 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:21.764 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:21.765 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.503662_155.1 2021-09-06T11:40:26.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:26.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:26.767 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:26.810 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:26.811 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:26.932 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:26.933 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:26.933 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:26.933 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:26.933 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:26.934 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:26.934 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:26.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:26.934 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:26.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:26.935 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:26.935 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:26.935 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:26.935 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:26.936 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:26.936 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:26.936 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:26.936 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:26.937 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:26.937 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:26.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:26.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:26.937 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:26.938 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:26.938 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:26.938 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:26.938 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:26.939 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:26.939 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:26.939 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:26.939 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:26.939 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:26.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:26.940 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:26.940 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:26.940 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:26.941 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:26.941 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:40:26.941 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:26.941 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:26.941 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:26.942 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:40:26.942 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:26.942 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:26.943 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:26.943 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:26.943 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:26.943 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:26.944 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:26.944 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:26.944 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:26.944 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:26.944 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:26.945 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:26.945 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:26.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:26.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:26.946 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:26.946 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:26.946 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:26.946 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:26.946 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:26.947 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:26.947 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:26.947 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.544174_161.1 2021-09-06T11:40:31.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:31.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:31.950 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:32.037 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:32.037 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:32.289 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:32.290 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:32.290 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:32.290 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:32.291 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:32.291 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:32.291 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:32.291 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:32.292 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:32.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:32.292 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:32.292 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:32.293 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:32.293 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:32.293 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:32.293 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:32.293 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:32.294 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:32.294 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:32.294 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:32.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:32.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:32.295 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:32.295 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:32.295 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:32.296 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:32.296 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:32.296 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:32.296 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:32.297 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:32.297 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:32.297 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:32.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:32.298 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:32.298 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:32.298 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:32.298 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:32.298 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:40:32.299 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:32.299 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:32.299 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:32.299 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:40:32.300 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:32.300 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:32.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:32.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:32.300 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:32.301 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:32.301 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:32.301 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:32.302 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:32.302 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:32.302 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:32.302 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:32.303 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:32.303 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:32.303 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:32.303 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:32.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:32.304 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:32.304 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:32.304 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:32.304 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:32.305 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:32.306 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:40:37.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:37.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:37.309 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:37.403 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:37.403 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:37.727 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:37.728 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:37.728 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:37.728 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:37.728 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:37.729 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:37.729 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:37.729 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:37.730 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:37.730 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:37.730 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:37.731 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:37.731 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:37.731 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:37.731 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:37.731 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:37.732 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:37.732 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:37.732 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:37.733 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:37.733 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:37.734 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:37.734 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:37.734 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:37.735 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:37.735 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:37.735 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:37.735 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:37.735 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:37.736 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:37.736 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:37.736 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:37.737 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:37.737 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:37.737 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:37.738 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:37.738 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:37.738 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:40:37.739 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:37.739 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:37.739 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:37.739 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:40:37.740 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:37.740 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:37.740 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:37.740 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:37.741 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:37.741 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:37.741 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:37.742 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:37.742 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:37.742 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:37.743 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:37.743 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:37.743 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:37.743 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:37.744 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:37.744 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:37.744 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:37.745 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:37.745 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:37.745 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:37.746 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:37.746 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:37.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:40:42.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:42.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:42.750 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:42.838 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:42.838 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:43.109 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:43.109 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:43.110 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:43.110 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:43.110 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:43.110 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:43.111 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:43.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:43.111 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:43.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:43.111 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:43.112 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:43.112 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:43.112 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:43.112 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:43.113 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:43.113 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:43.113 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:43.113 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:43.114 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:43.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:43.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:43.115 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:43.115 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:43.115 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:43.115 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:43.116 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:43.116 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:43.116 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:43.116 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:43.117 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:43.117 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:43.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:43.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:43.117 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:43.118 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:43.118 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:43.118 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:40:43.118 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:43.119 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:43.119 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:43.119 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:40:43.119 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:43.120 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:43.120 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:43.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:43.120 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:43.121 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:43.121 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:43.121 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:43.121 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:43.122 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:43.122 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:43.122 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:43.122 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:43.123 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:43.123 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:43.123 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:43.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:43.123 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:43.124 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:43.124 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:43.125 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:43.125 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:43.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:40:48.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:48.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:48.129 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:48.220 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:48.220 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:48.471 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:48.471 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:48.471 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:48.472 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:48.472 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:48.472 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:48.472 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:48.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:48.473 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:48.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:48.473 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:48.473 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:48.474 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:48.474 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:48.474 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:48.474 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:48.475 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:48.475 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:48.475 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:48.475 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:48.476 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:48.476 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:48.476 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:48.476 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:48.476 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:48.477 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:48.477 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:48.477 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:48.477 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:48.478 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:48.478 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:48.478 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:48.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:48.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:48.479 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:48.479 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:48.479 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:48.480 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.547334_162.1", 2021-09-06T11:40:48.480 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:48.480 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:48.480 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:48.480 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.547334Z", 2021-09-06T11:40:48.481 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:48.481 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:48.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:48.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:48.482 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:48.482 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:48.482 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:48.482 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:48.482 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:48.483 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:48.483 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:48.483 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:48.483 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:48.484 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:48.484 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:48.484 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:48.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:48.484 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:48.485 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:48.485 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:48.485 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:48.485 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:48.486 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.547334_162.1 2021-09-06T11:40:53.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:53.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:53.489 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:53.577 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:53.577 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:53.908 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:53.908 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:53.908 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:53.909 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:53.909 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:53.909 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:53.909 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:53.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:53.910 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:53.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:53.910 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:53.910 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:53.911 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:53.911 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:53.911 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:53.911 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:53.912 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:53.912 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:53.912 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:53.912 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:53.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:53.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:53.913 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:53.913 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:53.914 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:53.914 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:53.914 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:53.914 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:53.915 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:53.915 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:53.915 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:53.915 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:53.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:53.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:53.916 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:53.916 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:53.916 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:53.917 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:40:53.917 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:53.917 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:53.917 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:53.918 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:40:53.918 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:53.918 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:53.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:53.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:53.919 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:53.919 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:53.919 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:53.919 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:53.920 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:53.920 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:53.920 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:53.920 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:53.921 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:53.921 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:53.921 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:53.921 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:53.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:53.922 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:53.922 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:53.922 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:53.922 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:53.923 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:53.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.544174_161.1 2021-09-06T11:40:58.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:58.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:40:58.927 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:40:59.011 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:40:59.011 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:40:59.278 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:40:59.278 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:40:59.278 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:40:59.279 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:40:59.279 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:40:59.279 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:40:59.279 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:59.280 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:59.280 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:40:59.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:59.280 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:40:59.280 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:59.281 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:59.281 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:40:59.281 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:59.281 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:40:59.282 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:59.282 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:40:59.282 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:59.282 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:59.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:59.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:59.283 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:40:59.283 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:59.284 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:59.284 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:40:59.284 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:59.284 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:59.284 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:59.285 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:40:59.285 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:59.285 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:59.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:59.286 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:59.286 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:40:59.286 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:59.286 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:59.287 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.539819_159.1", 2021-09-06T11:40:59.287 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:59.287 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:59.287 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:59.288 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.539819Z", 2021-09-06T11:40:59.288 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:59.288 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:59.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:59.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:40:59.289 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:40:59.289 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:40:59.289 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:40:59.289 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:40:59.289 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:40:59.290 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:40:59.290 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:40:59.290 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:40:59.290 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:40:59.291 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:59.291 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:59.291 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:40:59.291 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:40:59.292 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:40:59.292 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:40:59.292 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:40:59.292 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:40:59.293 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:40:59.294 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.539819_159.1 2021-09-06T11:41:04.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:04.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:04.297 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:04.364 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:04.365 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:04.480 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:04.481 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:04.482 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:04.483 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:04.484 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:04.485 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:04.485 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:04.485 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:04.485 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:04.486 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:04.486 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:04.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:04.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:04.487 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:04.487 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:04.487 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:04.487 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:04.488 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:04.488 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:04.488 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:04.488 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:04.488 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:04.489 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:04.489 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:04.489 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:04.489 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:04.489 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:04.490 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:04.490 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:41:04.490 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:04.490 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:04.491 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:04.491 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:41:04.491 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:04.491 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:04.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:04.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:04.492 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:04.492 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:04.492 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:04.493 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:04.493 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:04.493 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:04.493 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:04.494 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:04.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:04.495 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:04.495 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:04.495 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:04.496 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:04.496 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:04.497 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.503662_155.1 2021-09-06T11:41:09.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:09.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:09.501 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:09.588 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:09.589 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:09.897 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:09.898 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:09.898 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:09.899 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:09.899 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:09.900 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:09.900 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:09.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:09.900 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:09.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:09.901 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:09.902 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:09.902 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:09.902 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:09.902 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:09.903 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:09.903 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:09.903 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:09.903 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:09.904 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:09.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:09.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:09.905 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:09.905 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:09.905 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:09.905 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:09.906 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:09.906 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:09.906 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:09.906 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:09.907 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:09.907 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:09.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:09.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:09.908 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:09.908 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:09.908 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:09.908 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:41:09.908 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:09.909 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:09.909 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:09.909 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:41:09.909 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:09.910 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:09.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:09.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:09.911 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:09.911 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:09.911 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:09.911 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:09.912 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:09.912 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:09.912 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:09.912 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:09.913 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:09.913 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:09.913 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:09.913 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:09.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:09.914 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:09.914 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:09.915 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:09.915 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:09.915 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:09.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:41:14.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:14.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:14.919 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:15.009 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:15.009 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:15.405 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:15.406 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:15.406 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:15.407 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:15.407 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:15.407 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:15.407 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:15.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:15.408 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:15.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:15.408 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:15.409 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:15.409 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:15.409 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:15.409 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:15.409 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:15.410 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:15.410 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:15.410 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:15.411 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:15.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:15.411 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:15.411 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:15.412 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:15.412 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:15.412 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:15.412 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:15.412 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:15.413 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:15.413 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:15.413 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:15.413 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:15.413 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:15.414 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:15.414 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:15.414 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:15.415 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:15.415 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:41:15.415 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:15.415 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:15.415 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:15.416 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:41:15.416 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:15.416 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:15.416 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:15.417 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:15.417 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:15.417 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:15.417 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:15.417 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:15.418 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:15.418 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:15.418 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:15.418 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:15.419 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:15.419 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:15.419 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:15.419 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:15.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:15.420 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:15.420 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:15.420 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:15.420 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:15.421 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:15.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.484822_152.1 2021-09-06T11:41:20.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:20.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:20.425 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:20.499 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:20.500 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:20.579 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:20.579 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:20.579 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:20.580 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:20.580 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:20.580 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:20.580 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:20.581 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:20.582 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:20.583 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:20.584 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:20.585 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:20.586 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:41:20.587 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:41:20.588 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:20.589 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:20.590 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:20.590 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:20.590 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:20.590 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:20.591 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:20.591 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:20.591 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:20.591 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:20.592 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:20.592 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:20.592 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:20.592 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:20.592 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:20.593 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:20.593 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:20.593 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:20.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.508454_157.1 2021-09-06T11:41:25.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:25.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:25.597 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:25.703 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:25.704 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:26.035 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:26.035 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:26.036 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:26.036 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:26.036 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:26.036 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:26.037 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:26.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:26.037 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:26.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:26.038 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:26.038 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:26.038 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:26.038 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:26.039 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:26.039 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:26.039 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:26.039 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:26.040 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:26.040 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:26.040 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:26.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:26.040 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:26.041 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:26.041 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:26.041 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:26.041 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:26.042 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:26.042 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:26.042 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:26.042 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:26.043 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:26.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:26.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:26.043 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:26.043 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:26.044 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:26.044 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:41:26.044 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:26.044 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:26.045 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:26.045 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:41:26.045 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:26.045 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:26.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:26.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:26.046 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:26.046 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:26.046 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:26.047 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:26.047 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:26.047 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:26.047 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:26.048 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:26.048 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:26.048 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:26.048 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:26.048 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:26.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:26.049 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:26.049 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:26.049 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:26.050 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:26.050 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:26.051 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:41:31.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:31.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:31.054 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:31.146 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:31.147 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:31.610 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:31.610 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:31.610 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:31.611 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:31.611 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:31.611 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:31.611 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:31.612 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:31.612 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:31.612 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:31.612 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:31.612 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:31.613 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:31.613 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:31.613 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:31.613 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:31.614 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:31.614 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:31.614 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:31.614 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:31.615 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:31.615 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:31.615 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:31.615 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:31.616 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:31.616 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:31.616 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:31.617 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:31.617 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:31.617 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:31.617 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:31.618 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:31.618 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:31.618 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:31.618 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:31.619 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:31.619 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:31.619 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:41:31.619 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:31.620 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:31.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:31.620 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:41:31.620 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:31.620 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:31.621 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:31.621 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:31.621 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:31.621 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:31.622 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:31.622 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:31.622 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:31.622 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:31.623 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:31.623 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:31.623 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:31.623 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:31.623 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:31.624 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:31.624 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:31.624 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:31.624 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:31.625 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:31.625 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:31.625 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:31.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.544174_161.1 2021-09-06T11:41:36.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:36.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:36.630 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:36.707 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:36.707 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:37.079 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:37.079 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:37.079 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:37.080 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:37.081 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:37.082 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:37.082 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:37.082 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:37.082 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:37.082 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:37.083 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:37.084 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:37.085 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:37.085 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:37.085 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:37.085 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:37.085 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:37.086 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:37.086 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:37.086 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:37.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:37.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:37.087 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:37.087 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:37.087 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:37.087 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:41:37.088 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:37.088 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:37.088 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:37.088 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:41:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:37.089 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:37.090 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:37.091 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:37.092 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:37.093 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:37.093 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:37.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.544174_161.1 2021-09-06T11:41:42.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:42.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:42.097 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:42.180 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:42.181 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:42.552 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:42.553 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:42.553 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:42.553 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:42.553 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:42.554 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:42.554 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:42.554 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:42.554 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:42.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:42.555 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:42.555 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:42.555 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:42.557 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:42.557 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:42.557 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:42.557 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:42.558 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:42.558 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:42.558 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:42.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:42.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:42.559 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:42.559 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:42.559 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:42.559 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:42.560 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:42.560 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:42.560 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:42.560 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:42.561 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:42.561 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:42.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:42.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:42.562 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:42.562 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:42.562 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:42.562 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:41:42.563 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:42.563 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:42.563 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:42.563 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:41:42.564 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:42.564 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:42.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:42.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:42.565 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:42.565 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:42.565 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:42.565 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:42.566 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:42.566 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:42.566 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:42.566 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:42.567 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:42.567 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:42.567 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:42.568 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:42.568 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:42.568 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:42.569 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:42.569 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:42.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:42.570 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:42.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.484822_152.1 2021-09-06T11:41:47.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:47.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:47.576 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:47.676 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:47.676 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:48.082 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:48.083 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:48.083 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:48.083 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:48.084 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:48.084 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:48.084 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:48.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:48.085 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:48.085 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:48.085 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:48.085 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:48.085 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:48.086 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:48.086 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:48.086 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:48.086 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:48.087 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:48.087 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:48.087 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:48.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:48.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:48.088 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:48.088 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:48.088 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:48.088 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:48.089 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:48.089 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:48.089 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:48.089 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:48.090 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:48.090 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:48.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:48.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:48.091 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:48.091 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:48.092 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:48.092 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.759827_175.1", 2021-09-06T11:41:48.092 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:48.092 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:48.093 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:48.093 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.759827Z", 2021-09-06T11:41:48.093 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:48.093 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:48.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:48.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:48.094 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:48.094 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:48.095 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:48.095 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:48.095 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:48.095 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:48.095 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:48.096 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:48.096 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:48.096 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:48.096 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:48.097 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:48.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:48.097 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:48.097 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:48.098 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:48.098 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:48.098 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:48.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.759827_175.1 2021-09-06T11:41:48.666 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:41:48.666+0000 7fc4b0ff6700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T11:41:48.777 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:41:48.776+0000 7fc4b2a0c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T11:41:53.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:53.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:53.101 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:53.150 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:53.150 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:53.632 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:53.632 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:53.632 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:53.632 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:53.633 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:53.633 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:53.633 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:53.634 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:53.634 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:53.634 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:53.634 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:53.635 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:53.635 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:53.635 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:53.636 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:53.636 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:53.636 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:53.636 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:53.636 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:53.637 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:53.637 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:53.637 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:53.637 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:53.638 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:53.638 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:53.638 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:53.638 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:53.639 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:53.640 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:53.640 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:53.640 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:53.640 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:53.641 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:53.641 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:41:53.641 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:53.641 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:53.642 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:53.642 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:41:53.642 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:53.642 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:53.643 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:53.643 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:53.643 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:53.643 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:53.643 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:53.644 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:53.644 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:53.644 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:53.644 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:53.645 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:53.645 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:53.645 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:53.645 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:53.646 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:53.646 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:53.646 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:53.646 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:53.647 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:53.647 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:53.647 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:53.647 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.508454_157.1 2021-09-06T11:41:58.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:58.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:41:58.650 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:41:58.749 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:41:58.749 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:41:59.100 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:41:59.100 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:41:59.101 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:41:59.101 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:41:59.101 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:41:59.101 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:41:59.102 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:59.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:59.102 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:41:59.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:59.103 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:41:59.103 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:59.103 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:59.103 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:41:59.104 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:59.104 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:41:59.104 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:59.104 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:41:59.104 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:59.105 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:59.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:59.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:59.105 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:41:59.106 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:59.106 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:59.106 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:41:59.106 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:59.106 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:59.107 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:59.107 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:41:59.107 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:59.107 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:59.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:59.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:59.108 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:41:59.108 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:59.108 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:59.109 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:41:59.109 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:59.109 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:59.109 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:59.109 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:41:59.110 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:59.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:59.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:59.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:41:59.111 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:41:59.111 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:41:59.111 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:41:59.111 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:41:59.111 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:41:59.112 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:41:59.112 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:41:59.112 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:41:59.112 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:41:59.112 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:59.113 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:59.113 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:41:59.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:41:59.113 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:41:59.114 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:41:59.114 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:41:59.114 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:41:59.114 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:41:59.115 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:42:04.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:04.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:04.118 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:04.205 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:04.205 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:04.532 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:04.533 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:04.533 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:04.533 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:04.534 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:04.534 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:04.534 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:04.534 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:04.535 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:04.535 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:04.535 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:04.535 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:04.536 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:04.536 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:04.536 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:04.536 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:04.536 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:04.537 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:04.537 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:04.537 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:04.537 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:04.538 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:04.538 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:04.538 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:04.538 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:04.538 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:04.539 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:04.539 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:04.539 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:04.539 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:04.540 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:04.541 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:04.541 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:04.541 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:42:04.541 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:04.541 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:04.542 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:04.542 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:42:04.542 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:04.542 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:04.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:04.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:04.543 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:04.543 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:04.544 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:04.544 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:04.544 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:04.545 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:04.545 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:04.546 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:04.546 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:04.546 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:04.547 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:04.547 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:04.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:04.547 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:04.548 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:04.548 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:04.548 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:04.549 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:04.551 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.484822_152.1 2021-09-06T11:42:09.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:09.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:09.554 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:09.633 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:09.634 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:09.827 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:09.827 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:09.827 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:09.828 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:09.828 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:09.828 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:09.829 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:09.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:09.829 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:09.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:09.830 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:09.830 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:09.830 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:09.830 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:09.831 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:09.831 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:09.831 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:09.831 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:09.831 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:09.832 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:09.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:09.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:09.832 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:09.832 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:09.833 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:09.833 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:09.833 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:09.833 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:09.834 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:09.834 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:09.834 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:09.834 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:09.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:09.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:09.835 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:09.835 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:09.835 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:09.836 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:42:09.836 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:09.836 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:09.836 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:09.837 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:42:09.837 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:09.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:09.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:09.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:09.838 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:09.838 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:09.838 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:09.838 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:09.839 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:09.839 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:09.839 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:09.839 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:09.839 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:09.840 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:09.840 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:09.840 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:09.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:09.841 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:09.841 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:09.841 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:09.841 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:09.842 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:09.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.544174_161.1 2021-09-06T11:42:14.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:14.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:14.845 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:14.931 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:14.931 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:15.286 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:15.287 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:15.287 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:15.287 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:15.288 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:15.288 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:15.288 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:15.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:15.288 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:15.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:15.289 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:15.289 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:15.289 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:15.290 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:15.290 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:15.290 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:15.290 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:15.291 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:15.291 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:15.291 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:15.291 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:15.291 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:15.292 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:15.292 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:15.292 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:15.292 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:15.293 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:15.293 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:15.293 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:15.293 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:15.294 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:15.294 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:15.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:15.294 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:15.294 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:15.295 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:15.295 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:15.295 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:42:15.295 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:15.295 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:15.296 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:15.296 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:42:15.296 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:15.296 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:15.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:15.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:15.297 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:15.297 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:15.298 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:15.298 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:15.298 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:15.298 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:15.299 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:15.299 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:15.299 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:15.299 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:15.299 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:15.300 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:15.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:15.300 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:15.300 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:15.301 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:15.301 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:15.301 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:15.302 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:42:20.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:20.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:20.306 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:20.407 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:20.408 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:20.773 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:20.774 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:20.774 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:20.774 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:20.775 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:20.775 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:20.775 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:20.775 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:20.775 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:20.776 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:20.776 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:20.776 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:20.776 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:20.777 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:20.777 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:20.777 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:20.777 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:20.778 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:20.778 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:20.778 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:20.778 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:20.779 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:20.779 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:20.779 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:20.779 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:20.779 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:20.780 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:20.780 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:20.780 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:20.781 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:20.781 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:20.781 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:20.781 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:20.782 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:20.782 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:20.782 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:20.782 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:20.783 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:42:20.783 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:20.783 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:20.783 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:20.784 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:42:20.784 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:20.784 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:20.784 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:20.785 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:20.785 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:20.785 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:20.785 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:20.786 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:20.786 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:20.786 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:20.786 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:20.787 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:20.787 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:20.787 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:20.787 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:20.788 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:20.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:20.788 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:20.788 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:20.788 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:20.789 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:20.789 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:20.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.738114_171.1 2021-09-06T11:42:25.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:25.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:25.792 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:25.890 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:25.890 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:26.198 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:26.199 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:26.199 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:26.199 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:26.200 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:26.200 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:26.200 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:26.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:26.201 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:26.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:26.201 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:26.201 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:26.202 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:26.202 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:26.202 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:26.202 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:26.202 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:26.203 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:26.203 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:26.203 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:26.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:26.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:26.204 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:26.204 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:26.204 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:26.204 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:26.204 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:26.205 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:26.205 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:26.205 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:26.205 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:26.206 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:26.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:26.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:26.206 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:26.206 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:26.207 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:26.207 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:42:26.207 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:26.207 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:26.207 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:26.208 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:42:26.208 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:26.208 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:26.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:26.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:26.209 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:26.209 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:26.209 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:26.209 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:26.210 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:26.210 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:26.210 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:26.210 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:26.211 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:26.211 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:26.211 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:26.211 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:26.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:26.212 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:26.212 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:26.212 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:26.212 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:26.213 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:26.214 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:42:31.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:31.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:31.216 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:31.303 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:31.304 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:31.647 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:31.647 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:31.648 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:31.648 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:31.649 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:31.649 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:31.650 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:31.650 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:31.650 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:31.651 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:31.651 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:31.651 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:31.651 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:31.652 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:31.652 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:31.652 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:31.652 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:31.652 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:31.653 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:31.653 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:31.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:31.653 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:31.654 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:31.654 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:31.654 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:31.654 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:31.654 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:31.655 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:31.655 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:31.655 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:31.655 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:31.656 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:31.656 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:31.656 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:31.656 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:31.656 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:31.657 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:31.657 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:42:31.657 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:31.657 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:31.658 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:31.658 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:42:31.658 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:31.658 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:31.658 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:31.659 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:31.659 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:31.659 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:31.659 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:31.660 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:31.660 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:31.660 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:31.661 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:31.661 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:31.662 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:31.662 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:31.663 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:31.663 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:31.664 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:31.664 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:31.664 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:31.665 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:31.665 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:31.666 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:31.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:42:36.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:36.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:36.671 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:36.752 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:36.753 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:36.988 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:36.988 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:36.989 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:36.989 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:36.989 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:36.989 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:36.990 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:36.990 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:36.990 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:36.991 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:36.991 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:36.991 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:36.991 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:36.991 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:36.992 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:36.992 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:36.992 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:36.992 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:36.993 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:36.993 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:36.994 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:36.994 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:36.994 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:36.994 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:36.995 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:36.995 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:36.995 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:36.996 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:36.996 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:36.996 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:36.996 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:36.997 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:36.997 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:36.997 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:36.997 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:36.997 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:36.998 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:36.998 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:42:36.998 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:36.998 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:36.999 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:36.999 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:42:36.999 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:36.999 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:36.999 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:37.000 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:37.000 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:37.000 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:37.000 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:37.001 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:37.001 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:37.001 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:37.002 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:37.002 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:37.002 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:37.002 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:37.003 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:37.003 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:37.003 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:37.004 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:37.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:42:42.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:42.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:42.008 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:42.109 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:42.110 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:42.319 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:42.319 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:42.320 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:42.320 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:42.320 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:42.321 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:42.321 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:42.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:42.321 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:42.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:42.322 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:42.322 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:42.322 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:42.323 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:42.323 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:42.323 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:42.323 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:42.323 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:42.324 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:42.324 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:42.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:42.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:42.325 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:42.325 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:42.325 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:42.325 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:42.326 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:42.326 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:42.326 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:42.326 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:42.326 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:42.327 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:42.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:42.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:42.327 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:42.327 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:42.328 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:42.328 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:42:42.328 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:42.328 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:42.329 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:42.329 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:42:42.329 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:42.329 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:42.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:42.330 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:42.330 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:42.330 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:42.330 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:42.330 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:42.331 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:42.331 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:42.331 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:42.331 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:42.332 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:42.332 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:42.332 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:42.332 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:42.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:42.333 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:42.333 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:42.333 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:42.333 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:42.333 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:42.335 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.484822_152.1 2021-09-06T11:42:47.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:47.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:47.337 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:47.401 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:47.402 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:47.821 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:47.821 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:47.822 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:47.822 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:47.823 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:47.823 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:47.823 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:47.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:47.824 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:47.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:47.825 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:47.826 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:47.826 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:47.827 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:47.827 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:47.827 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:47.828 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:47.828 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:47.828 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:47.828 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:47.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:47.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:47.829 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:47.829 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:47.830 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:47.830 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:47.830 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:47.830 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:47.831 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:47.831 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:47.831 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:47.831 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:47.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:47.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:47.832 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:47.832 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:47.833 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:47.833 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:42:47.833 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:47.833 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:47.834 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:47.834 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:42:47.834 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:47.834 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:47.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:47.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:47.835 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:47.835 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:47.836 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:47.836 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:47.836 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:47.836 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:47.837 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:47.837 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:47.837 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:47.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:47.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:47.838 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:47.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:47.838 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:47.838 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:47.839 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:47.839 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:47.839 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:47.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.544174_161.1 2021-09-06T11:42:52.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:52.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:52.843 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:52.891 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:52.891 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:53.149 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:53.149 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:53.149 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:53.150 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:53.150 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:53.150 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:53.150 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:53.151 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:53.151 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:53.151 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:53.151 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:53.152 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:53.152 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:53.152 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:53.152 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:53.152 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:53.153 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:53.153 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:53.153 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:53.153 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:53.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:53.154 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:53.154 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:53.154 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:53.155 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:53.155 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:53.155 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:53.155 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:53.155 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:53.156 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:53.156 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:53.156 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:53.156 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:53.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:53.157 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:53.157 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:53.157 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:53.157 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:42:53.158 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:53.158 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:53.158 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:53.158 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:42:53.159 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:53.159 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:53.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:53.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:53.159 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:53.160 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:53.160 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:53.160 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:53.160 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:53.161 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:53.161 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:53.161 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:53.161 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:53.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:53.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:53.162 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:53.162 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:53.162 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:53.163 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:53.163 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:53.163 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:53.163 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:53.164 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.503662_155.1 2021-09-06T11:42:58.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:58.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:42:58.166 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:42:58.219 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:42:58.219 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:42:58.294 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:42:58.364 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:42:58.365 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:42:58.365 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:42:58.365 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:42:58.366 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:42:58.366 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:58.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:58.366 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:42:58.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:58.367 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:42:58.367 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:58.367 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:58.368 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:42:58.368 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:58.368 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:42:58.368 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:58.369 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:42:58.369 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:58.369 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:58.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:58.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:58.370 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:42:58.370 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:58.370 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:58.370 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:42:58.371 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:58.371 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:58.371 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:58.371 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:42:58.371 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:58.372 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:58.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:58.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:58.372 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:42:58.372 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:58.373 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:58.373 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:42:58.373 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:58.373 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:58.374 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:58.374 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:42:58.374 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:58.374 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:58.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:58.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:42:58.375 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:42:58.375 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:42:58.375 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:42:58.376 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:42:58.376 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:42:58.376 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:42:58.376 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:42:58.377 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:42:58.377 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:42:58.377 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:58.377 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:58.377 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:42:58.378 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:42:58.378 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:42:58.378 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:42:58.378 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:42:58.378 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:42:58.379 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:42:58.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.508454_157.1 2021-09-06T11:43:03.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:03.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:03.383 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:03.427 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:03.427 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:03.495 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:03.496 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:03.497 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:03.498 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:03.499 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:03.500 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:03.501 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:03.502 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:03.502 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:03.502 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:03.502 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:03.503 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.543029_160.1", 2021-09-06T11:43:03.504 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.543029Z", 2021-09-06T11:43:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:03.505 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:03.506 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:03.507 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:03.508 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:03.509 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:03.510 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:03.510 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:03.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.543029_160.1 2021-09-06T11:43:08.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:08.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:08.514 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:08.557 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:08.557 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:08.630 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:08.630 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:08.630 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:08.631 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:08.631 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:08.631 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:08.631 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:08.631 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:08.632 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:08.632 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:08.632 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:08.632 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:08.633 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:08.633 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:08.633 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:08.633 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:08.633 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:08.634 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:08.634 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:08.634 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:08.634 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:08.635 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:08.635 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:08.635 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:08.635 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:08.636 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:08.636 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:08.636 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:08.636 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:08.636 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:08.637 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:08.637 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:08.637 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:08.637 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:08.638 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:08.638 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:08.638 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:08.638 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:43:08.638 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:08.639 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:08.639 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:08.639 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:43:08.639 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:08.640 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:08.640 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:08.640 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:08.640 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:08.640 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:08.641 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:08.641 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:08.641 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:08.641 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:08.642 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:08.642 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:08.642 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:08.642 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:08.643 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:08.643 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:08.643 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:08.643 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:08.644 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:08.644 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:08.644 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:08.644 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:08.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.507089_156.1 2021-09-06T11:43:13.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:13.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:13.648 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:13.744 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:13.744 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:14.055 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:14.055 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:14.055 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:14.056 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:14.056 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:14.056 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:14.056 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:14.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:14.057 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:14.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:14.057 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:14.058 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:14.058 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:14.058 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:14.058 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:14.058 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:14.059 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:14.059 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:14.059 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:14.059 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:14.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:14.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:14.060 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:14.060 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:14.061 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:14.061 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:14.061 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:14.061 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:14.062 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:14.062 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:14.062 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:14.062 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:14.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:14.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:14.063 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:14.063 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:14.063 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:14.063 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:43:14.064 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:14.064 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:14.064 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:14.064 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:43:14.065 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:14.065 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:14.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:14.065 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:14.065 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:14.066 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:14.066 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:14.066 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:14.066 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:14.067 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:14.067 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:14.067 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:14.067 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:14.067 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:14.068 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:14.068 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:14.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:14.068 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:14.069 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:14.069 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:14.069 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:14.069 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:14.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.484822_152.1 2021-09-06T11:43:19.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:19.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:19.074 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:19.155 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:19.156 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:19.539 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:19.539 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:19.540 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:19.540 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:19.540 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:19.540 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:19.541 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:19.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:19.541 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:19.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:19.541 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:19.542 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:19.542 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:19.542 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:19.543 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:19.543 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:19.543 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:19.543 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:19.544 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:19.544 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:19.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:19.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:19.544 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:19.545 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:19.545 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:19.545 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:19.545 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:19.546 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:19.546 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:19.546 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:19.546 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:19.547 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:19.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:19.547 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:19.547 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:19.547 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:19.548 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:19.548 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:43:19.548 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:19.548 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:19.548 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:19.549 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:43:19.549 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:19.549 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:19.549 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:19.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:19.550 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:19.550 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:19.550 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:19.551 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:19.551 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:19.551 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:19.551 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:19.551 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:19.552 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:19.552 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:19.552 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:19.552 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:19.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:19.553 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:19.553 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:19.553 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:19.554 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:19.554 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:19.554 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:43:24.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:24.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:24.558 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:24.648 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:24.649 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:24.892 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:24.892 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:24.892 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:24.892 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:24.893 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:24.893 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:24.893 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:24.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:24.894 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:24.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:24.894 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:24.894 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:24.895 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:24.895 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:24.895 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:24.895 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:24.895 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:24.896 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:24.896 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:24.896 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:24.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:24.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:24.897 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:24.897 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:24.897 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:24.897 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:24.898 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:24.898 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:24.898 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:24.898 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:24.898 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:24.899 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:24.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:24.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:24.899 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:24.900 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:24.900 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:24.900 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:43:24.900 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:24.900 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:43:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:24.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:24.902 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:24.902 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:24.902 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:24.902 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:24.903 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:24.903 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:24.903 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:24.903 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:24.903 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:24.904 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:24.904 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:24.904 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:24.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:24.905 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:24.905 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:24.905 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:24.905 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:24.906 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:24.907 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.508454_157.1 2021-09-06T11:43:29.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:29.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:29.910 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:29.959 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:29.959 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:30.192 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:30.192 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:30.192 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:30.193 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:30.193 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:30.193 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:30.193 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:30.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:30.194 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:30.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:30.194 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:30.195 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:30.195 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:30.195 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:30.195 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:43:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:30.202 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:43:30.202 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:30.202 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:30.204 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:30.204 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:30.204 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:30.204 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:30.204 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:30.205 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:30.205 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:30.205 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:30.205 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:30.206 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:30.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:30.206 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:30.206 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:30.206 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:30.207 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:30.207 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:30.207 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.503662_155.1 2021-09-06T11:43:35.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:35.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:35.210 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:35.267 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:35.267 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:35.491 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:35.491 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:35.491 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:35.492 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:35.492 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:35.492 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:35.492 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:35.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:35.493 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:35.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:35.493 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:35.493 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:35.494 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:35.494 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:35.494 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:35.494 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:35.495 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:35.495 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:35.495 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:35.495 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:35.496 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:35.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:35.496 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:35.496 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:35.496 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:35.497 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:35.497 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:35.497 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:35.497 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:35.497 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:35.498 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:35.498 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:35.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:35.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:35.498 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:35.499 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:35.499 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:35.499 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:43:35.499 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:35.500 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:35.500 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:35.500 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:43:35.500 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:35.500 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:35.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:35.501 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:35.501 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:35.501 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:35.501 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:35.502 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:35.502 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:35.502 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:35.502 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:35.503 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:35.503 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:35.503 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:35.503 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:35.503 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:35.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:35.504 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:35.504 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:35.504 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:35.505 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:35.505 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:35.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:43:40.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:40.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:40.508 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:40.847 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:40.847 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:41.166 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:41.166 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:41.166 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:41.167 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:41.167 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:41.167 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:41.167 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:41.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:41.168 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:41.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:41.168 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:41.168 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:41.169 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:41.169 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:41.169 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:41.169 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:41.170 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:41.170 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:41.170 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:41.170 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:41.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:41.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:41.171 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:41.171 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:41.172 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:41.172 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:41.172 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:41.172 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:41.173 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:41.173 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:41.173 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:41.173 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:41.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:41.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:41.174 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:41.174 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:41.174 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:41.175 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:43:41.175 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:41.175 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:41.175 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:41.176 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:43:41.176 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:41.176 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:41.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:41.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:41.177 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:41.177 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:41.177 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:41.178 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:41.178 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:41.178 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:41.178 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:41.179 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:41.179 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:41.179 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:41.179 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:41.180 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:41.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:41.180 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:41.180 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:41.181 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:41.181 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:41.181 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:41.182 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.508454_157.1 2021-09-06T11:43:46.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:46.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:46.185 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:46.258 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:46.259 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:46.637 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:46.637 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:46.637 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:46.638 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:46.638 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:46.638 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:46.638 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:46.639 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:46.639 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:46.639 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:46.639 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:46.640 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:46.640 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:46.640 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:46.640 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:46.640 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:46.641 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:46.641 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:46.641 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:46.641 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:46.641 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:46.642 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:46.642 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:46.642 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:46.642 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:46.643 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:46.643 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:46.643 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:46.643 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:46.643 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:46.644 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:46.644 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:46.644 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:46.644 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:46.644 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:46.645 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:46.645 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:46.645 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:43:46.645 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:46.646 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:46.646 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:46.646 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:43:46.646 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:46.647 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:46.647 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:46.647 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:46.647 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:46.648 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:46.648 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:46.648 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:46.648 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:46.648 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:46.649 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:46.649 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:46.649 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:46.649 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:46.650 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:46.650 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:46.650 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:46.650 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:46.651 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:46.651 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:46.651 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:46.651 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:46.652 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.544174_161.1 2021-09-06T11:43:51.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:51.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:51.654 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:51.715 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:51.716 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:52.131 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:52.131 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:52.132 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:52.132 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:52.132 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:52.132 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:52.132 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:52.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:52.133 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:52.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:52.133 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:52.133 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:52.134 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:52.134 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:52.134 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:52.134 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:52.135 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:52.135 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:52.135 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:52.135 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:52.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:52.136 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:52.136 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:52.136 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:52.136 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:52.137 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:52.137 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:52.137 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:52.137 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:52.137 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:52.138 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:52.138 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:52.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:52.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:52.139 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:52.139 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:52.139 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:52.139 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:43:52.140 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:52.140 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:52.140 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:52.140 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:43:52.140 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:52.141 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:52.141 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:52.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:52.141 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:52.141 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:52.142 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:52.142 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:52.142 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:52.142 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:52.143 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:52.143 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:52.143 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:52.143 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:52.143 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:52.144 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:52.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:52.144 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:52.144 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:52.145 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:52.145 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:52.145 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:52.145 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:43:57.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:57.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:43:57.148 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:43:57.207 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:43:57.208 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:43:57.540 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:43:57.540 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:43:57.540 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:43:57.541 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:43:57.541 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:43:57.541 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:43:57.541 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:57.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:57.542 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:43:57.542 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:57.542 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:43:57.543 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:57.543 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:57.543 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:43:57.543 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:57.543 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:43:57.544 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:57.544 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:43:57.544 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:57.544 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:57.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:57.545 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:57.545 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:43:57.545 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:57.546 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:57.546 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:43:57.546 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:57.546 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:57.547 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:57.547 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:43:57.547 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:57.547 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:57.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:57.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:57.548 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:43:57.548 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:57.548 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:57.549 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:43:57.549 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:57.549 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:57.549 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:57.549 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:43:57.550 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:57.550 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:57.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:57.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:43:57.551 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:43:57.551 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:43:57.551 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:43:57.551 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:43:57.551 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:43:57.552 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:43:57.552 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:43:57.552 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:43:57.552 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:43:57.553 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:57.553 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:57.553 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:43:57.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:43:57.554 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:43:57.554 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:43:57.554 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:43:57.554 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:43:57.554 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:43:57.555 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.508454_157.1 2021-09-06T11:44:02.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:02.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:02.558 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:02.610 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:02.611 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:02.831 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:44:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:44:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:02.846 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:02.847 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.511634_158.1 2021-09-06T11:44:07.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:07.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:07.850 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:07.945 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:07.946 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:08.198 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:08.199 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:08.199 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:08.199 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:08.199 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:08.200 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:08.200 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:08.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:08.200 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:08.205 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:08.205 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:08.205 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:08.205 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:08.206 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:08.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:08.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:08.206 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:44:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:08.208 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:08.208 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:44:08.208 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:08.208 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:08.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:08.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:08.209 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:08.209 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:08.210 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:08.210 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:08.210 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:08.210 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:08.212 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:08.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:08.212 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:08.212 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:08.213 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:08.213 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:08.213 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:08.214 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:44:13.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:13.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:13.217 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:13.307 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:13.307 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:13.616 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:13.616 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:13.617 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:13.617 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:13.617 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:13.617 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:13.618 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:13.618 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:13.618 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:13.618 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:13.619 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:13.619 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:13.619 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:13.619 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:13.620 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:13.620 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:13.620 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:13.620 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:13.621 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:13.621 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:13.621 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:13.621 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:13.622 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:13.622 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:13.622 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:13.622 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:13.622 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:13.623 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:13.623 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:13.623 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:13.623 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:13.624 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:13.624 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:13.624 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:13.624 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:13.624 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:13.625 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:13.625 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:44:13.625 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:13.625 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:13.626 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:13.626 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:44:13.626 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:13.626 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:13.627 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:13.627 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:13.627 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:13.627 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:13.628 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:13.628 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:13.628 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:13.628 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:13.628 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:13.629 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:13.629 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:13.629 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:13.629 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:13.630 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:13.630 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:13.630 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:13.630 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:13.631 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:13.631 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:13.631 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:13.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.508454_157.1 2021-09-06T11:44:18.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:18.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:18.635 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:18.717 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:18.717 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:19.139 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:19.139 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:19.140 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:19.140 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:19.140 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:19.140 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:19.141 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:19.141 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:19.141 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:19.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:19.141 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:19.142 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:19.142 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:19.142 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:19.142 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:19.143 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:19.143 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:19.143 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:19.143 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:19.144 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:19.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:19.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:19.144 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:19.144 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:19.145 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:19.145 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:19.145 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:19.145 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:19.146 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:19.146 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:19.146 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:19.146 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:19.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:19.147 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:19.147 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:19.147 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:19.147 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:19.148 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:44:19.148 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:19.148 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:19.148 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:19.148 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:44:19.149 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:19.149 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:19.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:19.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:19.150 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:19.150 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:19.150 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:19.150 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:19.150 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:19.151 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:19.151 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:19.151 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:19.151 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:19.152 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:19.152 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:19.152 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:19.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:19.153 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:19.153 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:19.153 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:19.153 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:19.154 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:19.155 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.508454_157.1 2021-09-06T11:44:24.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:24.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:24.158 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:24.262 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:24.263 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:24.591 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:24.592 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:24.592 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:24.592 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:24.592 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:24.593 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:24.593 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:24.593 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:24.593 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:24.594 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:24.594 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:24.594 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:24.594 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:24.595 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:24.595 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:24.595 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:24.595 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:24.596 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:24.596 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:24.596 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:24.596 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:24.596 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:24.597 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:24.597 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:24.597 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:24.597 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:24.598 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:24.598 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:24.598 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:24.598 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:24.599 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:24.599 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:24.599 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:24.599 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:24.599 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:24.600 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:24.600 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:24.600 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.539819_159.1", 2021-09-06T11:44:24.600 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:24.601 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:24.601 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:24.601 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.539819Z", 2021-09-06T11:44:24.601 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:24.602 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:24.602 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:24.602 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:24.603 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:24.603 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:24.603 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:24.603 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:24.603 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:24.604 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:24.604 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:24.604 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:24.604 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:24.605 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:24.605 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:24.605 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:24.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:24.606 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:24.606 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:24.606 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:24.606 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:24.607 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:24.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.539819_159.1 2021-09-06T11:44:29.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:29.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:29.611 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:29.708 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:29.708 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:30.191 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:30.191 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:30.191 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:30.192 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:30.192 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:30.192 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:30.192 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:30.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:30.193 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:30.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:30.193 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:30.194 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:30.194 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:30.194 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:30.195 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:30.195 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:30.195 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:30.195 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:30.195 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:30.196 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:30.197 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:30.198 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:30.199 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:30.200 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:44:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:44:30.201 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:30.202 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:30.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:30.203 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:30.204 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:30.204 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:30.204 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:30.205 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:30.205 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:30.205 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:30.205 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:30.205 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:30.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:30.206 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:30.206 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:30.206 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:30.207 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:30.207 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:30.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:44:35.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:35.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:35.211 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:35.300 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:35.300 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:35.792 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:35.792 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:35.792 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:35.792 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:35.793 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:35.793 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:35.793 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:35.793 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:35.793 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:35.794 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:35.794 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:35.794 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:35.794 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:35.795 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:35.795 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:35.795 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:35.795 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:35.796 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:35.796 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:35.796 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:35.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:35.797 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:35.797 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:35.797 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:35.797 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:35.797 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:35.798 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:35.798 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:35.798 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:35.798 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:35.799 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:35.799 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:35.799 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:35.799 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:35.800 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:35.800 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:35.800 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:35.800 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:44:35.801 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:35.801 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:35.801 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:35.801 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:44:35.801 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:35.802 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:35.802 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:35.802 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:35.802 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:35.803 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:35.803 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:35.803 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:35.803 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:35.803 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:35.804 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:35.804 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:35.804 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:35.804 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:35.805 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:35.805 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:35.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:35.805 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:35.806 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:35.806 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:35.806 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:35.806 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:35.807 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.544174_161.1 2021-09-06T11:44:40.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:40.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:40.810 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:40.877 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:40.877 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:41.461 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:41.461 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:41.461 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:41.462 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:41.462 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:41.462 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:41.462 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:41.463 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:41.463 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:41.463 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:41.463 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:41.463 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:41.464 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:41.464 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:41.464 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:41.464 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:41.465 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:41.465 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:41.465 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:41.465 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:41.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:41.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:41.466 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:41.466 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:41.467 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:41.467 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:41.467 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:41.467 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:41.467 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:41.468 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:41.468 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:41.468 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:41.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:41.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:41.469 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:41.469 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:41.469 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:41.469 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:44:41.470 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:41.470 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:41.470 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:41.470 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:44:41.470 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:41.471 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:41.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:41.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:41.471 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:41.472 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:41.472 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:41.472 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:41.472 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:41.472 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:41.473 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:41.473 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:41.473 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:41.473 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:41.474 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:41.474 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:41.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:41.474 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:41.474 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:41.475 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:41.475 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:41.475 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:41.475 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.484822_152.1 2021-09-06T11:44:46.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:46.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:46.479 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:46.573 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:46.574 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:47.051 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:47.052 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:47.052 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:47.052 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:47.052 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:47.053 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:47.053 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:47.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:47.053 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:47.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:47.054 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:47.054 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:47.054 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:47.054 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:47.055 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:47.055 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:47.055 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:47.055 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:47.056 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:47.056 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:47.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:47.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:47.056 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:47.057 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:47.057 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:47.057 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:47.057 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:47.058 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:47.058 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:47.058 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:47.058 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:47.059 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:47.059 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:47.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:47.059 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:47.059 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:47.060 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:47.060 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.547334_162.1", 2021-09-06T11:44:47.060 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:47.060 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:47.060 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:47.061 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.547334Z", 2021-09-06T11:44:47.061 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:47.061 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:47.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:47.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:47.062 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:47.062 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:47.062 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:47.062 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:47.063 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:47.063 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:47.063 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:47.063 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:47.063 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:47.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:47.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:47.064 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:47.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:47.065 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:47.065 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:47.065 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:47.065 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:47.065 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:47.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.547334_162.1 2021-09-06T11:44:52.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:52.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:52.069 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:52.134 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:52.134 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:52.695 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:52.695 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:52.695 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:52.696 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:52.696 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:52.696 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:52.696 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:52.696 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:52.697 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:52.697 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:52.697 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:52.697 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:52.698 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:52.698 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:52.698 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:52.698 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:52.699 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:52.699 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:52.699 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:52.699 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:52.700 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:52.700 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:52.700 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:52.700 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:52.701 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:52.701 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:52.701 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:52.702 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:52.702 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:52.702 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:52.702 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:52.703 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:52.703 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:52.703 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:52.703 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:52.703 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:52.704 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:52.704 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.798549_179.1", 2021-09-06T11:44:52.704 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:52.704 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:52.705 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:52.705 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.798549Z", 2021-09-06T11:44:52.705 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:52.705 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:52.706 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:52.706 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:52.706 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:52.706 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:52.707 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:52.707 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:52.707 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:52.707 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:52.707 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:52.708 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:52.708 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:52.708 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:52.708 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:52.709 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:52.709 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:52.709 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:52.709 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:52.709 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:52.710 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:52.710 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:52.710 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.798549_179.1 2021-09-06T11:44:57.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:57.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:44:57.713 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:44:57.804 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:44:57.805 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:44:58.256 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:44:58.257 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:44:58.257 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:44:58.257 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:44:58.257 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:44:58.258 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:44:58.258 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:58.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:58.258 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:44:58.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:58.259 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:44:58.259 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:58.259 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:58.260 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:44:58.260 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:58.260 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:44:58.260 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:58.261 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:44:58.261 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:58.261 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:58.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:58.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:58.262 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:44:58.262 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:58.262 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:58.262 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:44:58.263 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:58.263 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:58.263 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:58.263 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:44:58.264 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:58.264 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:58.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:58.264 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:58.265 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:44:58.265 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:58.265 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:58.265 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:44:58.265 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:58.266 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:58.266 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:58.266 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:44:58.266 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:58.267 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:58.267 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:58.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:44:58.268 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:44:58.268 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:44:58.269 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:44:58.269 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:44:58.269 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:44:58.270 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:44:58.270 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:44:58.270 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:44:58.271 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:44:58.271 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:58.271 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:58.271 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:44:58.272 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:44:58.272 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:44:58.272 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:44:58.272 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:44:58.272 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:44:58.273 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:44:58.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.484822_152.1 2021-09-06T11:45:03.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:03.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:03.276 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:03.356 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:03.357 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:03.734 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:03.735 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:03.735 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:03.735 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:03.735 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:03.736 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:03.736 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:03.736 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:03.736 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:03.737 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:03.737 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:03.737 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:03.737 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:03.737 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:03.738 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:03.738 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:03.738 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:03.738 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:03.739 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:03.739 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:03.739 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:03.739 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:03.740 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:03.740 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:03.740 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:03.740 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:03.740 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:03.741 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:03.741 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:03.741 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:03.741 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:03.742 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:03.742 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:03.742 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:03.742 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:03.743 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:03.743 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:03.743 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:45:03.743 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:03.743 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:03.744 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:03.744 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:45:03.744 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:03.744 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:03.745 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:03.745 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:03.745 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:03.745 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:03.745 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:03.746 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:03.746 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:03.746 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:03.746 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:03.747 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:03.747 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:03.747 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:03.748 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:03.748 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:03.748 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:03.748 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:03.749 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:03.749 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:03.749 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:03.749 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:03.750 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.484822_152.1 2021-09-06T11:45:08.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:08.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:08.753 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:08.837 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:08.837 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:09.049 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:09.050 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:09.050 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:09.051 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:09.051 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:09.051 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:09.051 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:09.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:09.052 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:09.052 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:09.052 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:09.053 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:09.053 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:09.053 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:09.053 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:09.053 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:09.054 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:09.054 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:09.054 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:09.054 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:09.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:09.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:09.055 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:09.055 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:09.056 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:09.056 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:09.056 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:09.056 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:09.056 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:09.057 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:09.057 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:09.057 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:09.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:09.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:09.058 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:09.058 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:09.058 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:09.059 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:45:09.059 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:09.059 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:09.059 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:09.060 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:45:09.060 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:09.060 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:09.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:09.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:09.061 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:09.061 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:09.061 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:09.061 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:09.062 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:09.062 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:09.062 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:09.062 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:09.062 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:09.063 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:09.063 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:09.063 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:09.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:09.064 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:09.064 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:09.064 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:09.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:09.064 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:09.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:45:14.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:14.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:14.068 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:14.145 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:14.145 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:14.390 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:14.390 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:14.390 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:14.391 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:14.391 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:14.391 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:14.391 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:14.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:14.392 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:14.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:14.392 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:14.393 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:14.393 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:14.393 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:14.393 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:14.393 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:14.394 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:14.394 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:14.394 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:14.394 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:14.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:14.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:14.395 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:14.395 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:14.395 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:14.396 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:14.396 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:14.396 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:14.396 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:14.397 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:14.397 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:14.397 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:14.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:14.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:14.398 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:14.398 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:14.398 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:14.398 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:45:14.399 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:14.399 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:14.399 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:14.399 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:45:14.400 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:14.400 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:14.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:14.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:14.400 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:14.401 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:14.401 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:14.401 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:14.401 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:14.402 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:14.402 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:14.402 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:14.402 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:14.402 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:14.403 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:14.403 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:14.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:14.403 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:14.404 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:14.404 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:14.404 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:14.404 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:14.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:45:19.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:19.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:19.408 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:19.498 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:19.498 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:19.930 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:19.931 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:19.931 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:19.931 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:19.932 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:19.932 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:19.932 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:19.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:19.933 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:19.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:19.933 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:19.934 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:19.934 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:19.934 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:19.934 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:19.935 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:19.935 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:19.935 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:19.935 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:19.936 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:19.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:19.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:19.936 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:19.937 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:19.937 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:19.937 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:19.937 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:19.938 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:19.938 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:19.938 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:19.938 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:19.939 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:19.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:19.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:19.939 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:19.940 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:19.940 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:19.940 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:45:19.940 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:19.941 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:19.941 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:19.941 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:45:19.941 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:19.942 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:19.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:19.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:19.942 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:19.943 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:19.943 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:19.943 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:19.943 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:19.944 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:19.944 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:19.944 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:19.944 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:19.944 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:19.945 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:19.945 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:19.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:19.945 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:19.946 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:19.946 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:19.946 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:19.946 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:19.948 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:45:24.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:24.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:24.950 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:25.015 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:25.016 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:25.308 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:25.308 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:25.308 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:25.309 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:25.309 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:25.309 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:25.309 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:25.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:25.310 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:25.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:25.310 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:25.310 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:25.311 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:25.311 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:25.311 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:25.311 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:25.312 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:25.312 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:25.312 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:25.312 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:25.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:25.313 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:25.313 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:25.313 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:25.313 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:25.313 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:25.314 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:25.314 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:25.314 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:25.314 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:25.315 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:25.315 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:25.315 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:25.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:25.315 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:25.316 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:25.316 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:25.316 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:45:25.316 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:25.317 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:25.317 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:25.317 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:45:25.317 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:25.317 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:25.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:25.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:25.318 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:25.318 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:25.318 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:25.319 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:25.319 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:25.319 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:25.319 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:25.320 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:25.320 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:25.320 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:25.320 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:25.320 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:25.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:25.321 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:25.321 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:25.321 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:25.322 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:25.322 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:25.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.486035_153.1 2021-09-06T11:45:30.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:30.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:30.326 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:30.421 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:30.422 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:30.880 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:30.881 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:30.881 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:30.881 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:30.881 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:30.882 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:30.882 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:30.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:30.882 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:30.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:30.883 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:30.883 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:30.883 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:30.883 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:30.884 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:30.884 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:30.885 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:30.885 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:30.885 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:30.885 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:30.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:30.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:30.886 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:30.886 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:30.887 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:30.887 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:30.887 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:30.888 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:30.888 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:30.888 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:30.888 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:30.889 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:30.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:30.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:30.889 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:30.889 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:30.890 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:30.890 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:45:30.890 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:30.890 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:30.891 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:30.891 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:45:30.891 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:30.891 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:30.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:30.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:30.892 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:30.892 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:30.892 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:30.893 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:30.893 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:30.893 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:30.893 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:30.893 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:30.894 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:30.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:30.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:30.894 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:30.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:30.895 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:30.895 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:30.895 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:30.896 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:30.896 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:30.897 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.503662_155.1 2021-09-06T11:45:35.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:35.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:35.900 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:35.974 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:35.974 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:36.237 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:36.237 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:36.237 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:36.237 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:36.238 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:36.238 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:36.238 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:36.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:36.239 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:36.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:36.239 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:36.239 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:36.239 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:36.240 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:36.240 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:36.240 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:36.240 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:36.241 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:36.241 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:36.241 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:36.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:36.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:36.242 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:36.242 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:36.242 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:36.242 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:36.243 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:36.243 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:36.243 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:36.243 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:36.244 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:36.244 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:36.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:36.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:36.245 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:36.245 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:36.245 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:36.245 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.539819_159.1", 2021-09-06T11:45:36.246 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:36.246 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:36.246 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:36.246 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.539819Z", 2021-09-06T11:45:36.247 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:36.247 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:36.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:36.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:36.247 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:36.248 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:36.248 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:36.248 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:36.248 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:36.248 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:36.249 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:36.249 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:36.249 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:36.249 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:36.249 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:36.250 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:36.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:36.250 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:36.250 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:36.251 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:36.251 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:36.251 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:36.252 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928415.920208_237.1 target=1_1630928091.539819_159.1 2021-09-06T11:45:41.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2021-09-06T11:45:41.255 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-42 2021-09-06T11:45:41.391 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-43 2021-09-06T11:45:41.644 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-44 2021-09-06T11:45:41.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:41.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:41.828 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:41.855 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:45:41.854+0000 7fc4be223700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:45:41.905 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:41.905 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:41.917 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:45:41.915+0000 7fc4bf225700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:45:41.955 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2021-09-06T11:45:41.850+0000 7f3e562be700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:45:41.956 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2021-09-06T11:45:41.855+0000 7f1430688700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:45:41.958 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2021-09-06T11:45:41.923+0000 7f3e542ba700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:45:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:45:42.251 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2021-09-06T11:45:42.250+0000 7f143268c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:45:42.294 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:45:42.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:45:42.295 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:42.295 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:45:42.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:45:42.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:45:42.296 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:45:42.296 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:45:42.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:45:42.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:45:42.297 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630928741.670299_257.1", 2021-09-06T11:45:42.297 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:45:41.670299Z" 2021-09-06T11:45:42.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:45:42.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:45:42.297 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:42.298 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:45:42.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:45:42.298 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:45:42.299 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:45:42.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:42.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:42.300 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:42.355 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:42.355 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:42.433 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:42.434 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:42.434 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:42.434 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:42.434 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:42.435 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:42.435 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:42.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:42.435 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:42.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:42.436 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:42.436 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:42.436 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:42.436 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:42.437 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:42.437 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:42.437 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:42.437 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:42.438 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:42.439 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:42.441 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:45:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:42.442 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:45:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:42.443 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:42.444 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:42.445 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:42.446 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:42.447 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:42.448 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:42.449 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.484822_152.1 2021-09-06T11:45:47.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:47.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:47.451 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:47.533 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:47.533 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:47.921 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:47.921 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:47.922 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:47.922 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:47.922 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:47.922 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:47.923 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:47.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:47.923 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:47.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:47.923 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:47.924 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:47.924 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:47.924 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:47.924 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:47.925 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:47.925 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:47.925 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:47.925 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:47.926 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:47.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:47.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:47.926 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:47.927 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:47.927 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:47.927 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:47.927 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:47.927 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:47.928 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:47.928 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:47.928 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:47.928 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:47.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:47.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:47.929 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:47.929 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:47.929 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:47.930 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:45:47.930 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:47.930 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:47.930 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:47.931 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:45:47.931 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:47.931 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:47.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:47.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:47.932 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:47.932 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:47.932 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:47.932 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:47.933 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:47.933 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:47.933 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:47.933 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:47.933 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:47.934 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:47.934 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:47.934 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:47.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:47.935 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:47.935 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:47.935 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:47.935 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:47.936 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:47.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.484822_152.1 2021-09-06T11:45:52.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:52.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:52.940 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:53.043 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:53.043 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:53.316 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:53.317 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:53.318 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:53.319 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:53.320 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:53.321 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:53.322 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:53.323 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:53.324 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:45:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:53.325 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:45:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:53.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:53.327 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:53.328 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:53.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:53.330 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:53.330 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:53.332 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.503662_155.1 2021-09-06T11:45:58.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:58.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:45:58.334 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:45:58.401 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:45:58.402 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:45:58.915 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:45:58.916 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:45:58.916 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:45:58.916 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:45:58.917 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:45:58.917 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:45:58.917 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:58.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:58.918 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:45:58.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:58.918 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:45:58.918 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:58.919 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:58.919 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:45:58.919 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:58.919 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:45:58.920 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:58.920 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:45:58.920 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:58.920 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:58.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:58.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:58.921 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:45:58.921 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:58.922 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:58.922 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:45:58.922 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:58.922 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:58.923 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:58.923 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:45:58.923 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:58.923 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:58.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:58.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:58.924 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:45:58.924 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:58.925 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:58.925 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:45:58.925 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:58.925 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:58.926 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:58.926 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:45:58.926 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:58.926 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:58.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:58.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:45:58.927 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:45:58.927 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:45:58.928 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:45:58.928 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:45:58.928 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:45:58.928 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:45:58.929 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:45:58.929 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:45:58.929 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:45:58.929 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:58.930 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:58.930 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:45:58.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:45:58.930 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:45:58.931 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:45:58.931 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:45:58.931 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:45:58.931 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:45:58.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.508454_157.1 2021-09-06T11:46:03.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:03.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:03.935 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:03.984 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:03.984 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:04.059 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:04.059 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:04.060 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:04.060 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:04.060 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:04.060 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:04.061 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:04.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:04.061 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:04.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:04.061 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:04.062 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:04.062 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:04.062 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:04.062 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:04.063 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:04.063 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:04.063 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:04.063 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:04.063 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:04.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:04.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:04.064 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:04.064 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:04.065 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:04.065 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:04.065 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:04.065 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:04.066 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:04.066 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:04.066 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:04.066 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:04.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:04.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:04.067 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:04.067 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:04.067 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:04.068 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:46:04.068 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:04.068 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:04.068 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:04.068 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:46:04.069 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:04.069 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:04.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:04.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:04.070 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:04.070 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:04.070 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:04.070 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:04.071 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:04.071 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:04.071 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:04.071 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:04.071 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:04.072 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:04.072 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:04.072 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:04.072 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:04.073 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:04.073 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:04.073 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:04.073 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:04.073 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:04.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.738114_171.1 2021-09-06T11:46:09.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:09.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:09.077 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:09.121 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:09.121 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:09.187 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:09.187 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:09.187 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:09.187 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:09.188 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:09.188 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:09.188 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:09.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:09.189 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:09.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:09.189 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:09.189 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:09.190 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:09.190 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:09.190 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:09.190 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:09.191 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:09.191 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:09.191 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:09.191 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:09.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:09.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:09.192 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:09.192 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:09.193 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:09.193 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:09.193 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:09.193 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:09.193 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:09.194 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:09.194 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:09.194 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:09.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:09.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:09.195 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:09.195 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:09.195 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:09.195 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:46:09.196 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:09.196 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:09.196 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:09.196 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:46:09.196 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:09.197 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:09.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:09.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:09.197 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:09.197 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:09.198 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:09.198 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:09.198 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:09.198 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:09.199 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:09.199 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:09.199 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:09.199 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:09.200 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:09.200 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:09.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:09.200 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:09.200 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:09.201 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:09.201 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:09.201 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:09.202 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.503662_155.1 2021-09-06T11:46:14.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:14.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:14.205 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:14.282 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:14.282 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:14.759 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:14.759 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:14.759 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:14.759 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:14.760 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:14.760 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:14.760 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:14.760 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:14.761 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:14.761 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:14.761 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:14.762 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:14.762 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:14.762 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:14.762 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:14.763 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:14.763 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:14.763 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:14.763 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:14.763 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:14.764 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:14.764 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:14.764 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:14.764 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:14.765 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:14.765 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:14.765 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:14.765 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:14.765 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:14.766 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:14.766 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:14.766 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:14.766 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:14.767 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:14.767 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:14.767 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:14.767 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:14.767 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:46:14.768 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:14.768 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:14.768 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:14.768 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:46:14.768 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:14.769 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:14.769 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:14.769 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:14.769 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:14.770 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:14.770 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:14.770 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:14.770 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:14.770 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:14.771 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:14.771 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:14.771 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:14.771 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:14.771 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:14.772 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:14.772 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:14.772 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:14.772 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:14.773 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:14.773 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:14.773 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:14.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.544174_161.1 2021-09-06T11:46:19.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:19.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:19.776 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:19.857 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:19.857 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:19.974 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:19.974 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:19.975 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:19.975 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:19.975 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:19.975 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:19.975 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:19.976 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:19.976 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:19.976 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:19.976 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:19.977 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:19.977 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:19.977 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:19.977 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:19.977 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:19.978 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:19.978 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:19.978 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:19.978 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:19.979 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:19.979 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:19.979 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:19.979 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:19.979 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:19.980 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:19.980 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:19.980 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:19.980 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:19.980 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:19.981 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:19.981 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:19.981 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:19.981 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:19.981 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:19.982 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:19.982 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:19.982 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.539819_159.1", 2021-09-06T11:46:19.982 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:19.983 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:19.983 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:19.983 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.539819Z", 2021-09-06T11:46:19.983 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:19.983 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:19.984 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:19.984 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:19.984 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:19.984 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:19.984 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:19.985 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:19.985 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:19.985 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:19.985 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:19.985 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:19.986 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:19.986 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:19.986 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:19.986 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:19.987 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:19.987 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:19.987 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:19.987 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:19.987 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:19.988 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:19.989 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.539819_159.1 2021-09-06T11:46:24.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:24.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:24.991 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:25.053 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:25.053 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:25.622 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:25.622 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:25.622 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:25.622 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:25.623 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:25.623 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:25.623 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:25.623 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:25.624 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:25.624 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:25.624 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:25.624 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:25.624 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:25.625 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:25.625 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:25.625 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:25.625 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:25.626 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:25.626 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:25.626 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:25.626 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:25.626 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:25.627 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:25.627 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:25.627 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:25.627 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:25.627 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:25.628 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:25.628 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:25.628 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:25.628 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:25.629 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:25.629 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:25.629 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:25.629 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:25.629 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:25.630 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:25.630 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:46:25.630 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:25.630 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:25.630 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:25.631 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:46:25.631 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:25.631 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:25.631 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:25.631 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:25.632 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:25.632 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:25.632 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:25.632 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:25.632 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:25.633 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:25.633 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:25.633 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:25.633 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:25.633 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:25.634 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:25.634 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:25.634 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:25.634 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:25.634 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:25.635 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:25.635 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:25.635 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:25.636 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.484822_152.1 2021-09-06T11:46:30.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:30.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:30.638 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:30.684 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:30.684 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:30.758 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:30.759 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:30.759 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:30.759 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:30.759 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:30.759 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:30.760 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:30.760 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:30.760 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:30.760 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:30.761 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:30.761 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:30.761 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:30.761 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:30.761 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:30.762 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:30.762 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:30.762 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:30.762 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:30.763 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:30.763 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:30.763 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:30.763 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:30.764 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:30.764 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:30.764 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:30.764 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:30.764 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:30.765 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:30.765 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:30.765 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:30.765 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:30.765 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:30.766 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:30.766 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:30.766 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:30.766 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:30.767 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.670393_167.1", 2021-09-06T11:46:30.767 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:30.767 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:30.768 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:30.768 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.670393Z", 2021-09-06T11:46:30.768 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:30.768 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:30.769 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:30.769 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:30.769 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:30.769 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:30.769 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:30.770 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:30.770 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:30.770 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:30.770 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:30.770 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:30.771 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:30.771 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:30.771 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:30.771 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:30.772 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:30.772 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:30.772 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:30.772 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:30.772 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:30.773 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:30.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.670393_167.1 2021-09-06T11:46:35.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:35.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:35.776 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:35.820 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:35.821 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:35.887 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:35.887 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:35.888 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:35.888 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:35.888 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:35.888 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:35.888 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:35.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:35.889 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:35.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:35.889 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:35.889 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:35.890 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:35.890 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:35.890 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:35.890 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:35.891 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:35.891 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:35.891 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:35.891 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:35.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:35.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:35.892 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:35.892 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:35.892 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:35.893 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:35.893 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:35.893 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:35.893 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:35.893 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:35.894 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:35.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:35.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:35.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:35.895 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:35.895 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:35.895 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:35.895 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T11:46:35.896 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:35.896 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:35.896 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:35.896 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T11:46:35.897 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:35.897 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:35.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:35.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:35.898 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:35.898 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:35.898 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:35.898 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:35.898 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:35.899 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:35.899 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:35.899 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:35.899 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:35.900 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:35.900 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:35.900 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:35.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:35.901 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:35.901 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:35.901 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:35.901 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:35.901 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:35.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.489232_154.1 2021-09-06T11:46:40.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:40.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:40.906 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:40.987 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:40.987 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:41.343 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:41.344 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:41.344 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:41.344 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:41.345 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:41.345 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:41.345 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:41.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:41.345 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:41.346 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:41.346 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:41.346 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:41.346 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:41.347 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:41.347 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:41.347 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:41.347 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:41.347 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:41.348 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:41.348 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:41.348 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:41.348 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:41.348 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:41.349 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:41.349 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:41.349 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:41.349 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:41.350 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:41.350 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:41.350 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:41.350 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:41.350 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:41.351 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:41.351 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:41.351 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:41.351 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:41.351 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:41.352 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.793863_177.1", 2021-09-06T11:46:41.352 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:41.352 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:41.352 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:41.353 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.793863Z", 2021-09-06T11:46:41.353 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:41.353 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:41.353 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:41.354 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:41.354 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:41.354 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:41.354 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:41.355 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:41.355 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:41.355 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:41.355 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:41.356 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:41.356 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:41.356 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:41.356 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:41.356 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:41.357 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:41.357 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:41.357 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:41.357 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:41.358 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:41.358 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:41.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.793863_177.1 2021-09-06T11:46:46.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:46.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:46.361 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:46.406 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:46.407 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:46.473 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:46:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:46.482 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:46:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:46.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:46.484 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:46.485 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:46.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:46.487 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:46.487 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:46.488 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.507089_156.1 2021-09-06T11:46:51.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:51.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:51.491 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:51.584 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:51.584 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:52.097 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:52.097 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:52.097 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:52.098 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:52.098 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:52.098 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:52.098 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:52.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:52.099 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:52.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:52.099 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:52.100 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:52.100 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:52.100 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:52.100 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:52.100 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:52.101 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:52.101 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:52.101 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:52.101 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:52.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:52.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:52.102 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:52.102 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:52.102 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:52.103 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:52.103 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:52.103 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:52.103 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:52.104 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:52.104 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:52.104 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:52.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:52.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:52.105 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:52.105 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:52.105 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:52.105 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:46:52.105 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:52.106 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:52.106 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:52.106 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:46:52.106 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:52.107 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:52.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:52.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:52.107 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:52.108 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:52.108 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:52.108 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:52.108 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:52.108 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:52.109 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:52.109 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:52.109 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:52.109 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:52.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:52.110 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:52.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:52.110 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:52.110 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:52.111 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:52.111 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:52.111 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:52.111 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.511634_158.1 2021-09-06T11:46:57.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:57.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:46:57.114 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:46:57.191 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:46:57.192 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:46:57.710 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:46:57.711 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:46:57.711 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:46:57.711 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:46:57.712 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:46:57.712 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:46:57.712 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:57.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:57.712 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:46:57.713 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:57.713 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:46:57.713 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:57.713 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:57.714 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:46:57.714 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:57.714 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:46:57.714 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:57.715 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:46:57.715 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:57.715 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:57.715 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:57.715 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:57.716 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:46:57.716 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:57.716 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:57.716 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:46:57.717 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:57.717 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:57.717 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:57.717 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:46:57.717 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:57.718 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:57.718 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:57.718 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:57.718 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:46:57.719 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:57.719 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:57.719 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:46:57.719 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:57.720 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:57.720 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:57.720 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:46:57.720 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:57.720 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:57.721 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:57.721 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:46:57.721 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:46:57.721 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:46:57.722 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:46:57.722 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:46:57.722 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:46:57.722 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:46:57.723 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:46:57.723 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:46:57.723 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:46:57.723 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:57.723 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:57.724 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:46:57.724 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:46:57.724 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:46:57.724 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:46:57.725 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:46:57.725 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:46:57.725 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:46:57.725 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.484822_152.1 2021-09-06T11:47:02.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:02.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:02.728 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:02.771 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:02.772 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:02.845 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:02.846 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:02.847 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:02.849 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:02.850 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:02.851 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:47:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:47:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:02.860 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:02.861 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.508454_157.1 2021-09-06T11:47:07.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:07.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:07.864 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:07.932 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:07.933 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:08.056 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:08.056 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:08.056 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:08.057 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:08.057 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:08.057 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:08.057 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:08.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:08.058 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:08.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:08.058 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:08.059 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:08.059 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:08.059 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:08.059 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:08.060 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:08.061 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:08.062 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:08.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:08.064 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:47:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:08.065 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:47:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:08.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:08.067 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:08.068 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:08.069 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:08.070 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:08.071 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.486035_153.1 2021-09-06T11:47:13.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:13.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:13.074 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:13.148 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:13.148 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:13.266 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:13.266 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:13.267 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:13.267 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:13.267 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:13.267 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:13.267 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:13.268 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:13.268 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:13.268 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:13.268 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:13.269 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:13.269 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:13.269 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:13.269 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:13.269 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:13.270 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:13.270 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:13.270 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:13.270 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:13.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:13.271 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:13.271 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:13.271 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:13.271 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:13.272 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:13.272 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:13.272 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:13.272 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:13.273 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:13.273 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:13.274 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:13.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:13.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:13.274 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:13.275 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:13.275 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:13.275 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928092.007542_189.1", 2021-09-06T11:47:13.275 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:13.276 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:13.276 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:13.276 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:52.007542Z", 2021-09-06T11:47:13.276 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:13.276 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:13.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:13.277 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:13.277 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:13.277 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:13.277 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:13.278 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:13.278 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:13.278 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:13.278 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:13.279 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:13.279 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:13.279 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:13.279 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:13.279 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:13.280 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:13.280 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:13.280 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:13.280 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:13.280 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:13.281 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:13.282 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928092.007542_189.1 2021-09-06T11:47:18.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:18.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:18.284 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:18.373 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:18.373 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:18.857 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:18.857 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:18.857 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:18.858 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:18.858 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:18.858 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:18.858 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:18.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:18.859 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:18.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:18.859 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:18.859 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:18.860 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:18.860 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:18.860 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:18.860 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:18.860 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:18.861 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:18.861 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:18.861 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:18.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:18.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:18.862 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:18.862 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:18.862 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:18.862 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:18.863 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:18.863 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:18.863 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:18.863 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:18.863 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:18.864 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:18.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:18.864 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:18.864 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:18.865 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:18.865 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:18.865 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:47:18.865 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:18.865 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:18.866 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:18.866 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:47:18.866 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:18.866 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:18.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:18.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:18.867 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:18.867 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:18.867 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:18.868 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:18.868 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:18.868 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:18.868 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:18.868 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:18.869 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:18.869 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:18.869 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:18.869 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:18.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:18.870 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:18.870 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:18.870 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:18.870 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:18.870 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:18.872 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.503662_155.1 2021-09-06T11:47:23.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:23.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:23.875 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:23.944 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:23.944 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:24.556 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:24.557 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:24.557 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:24.557 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:24.557 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:24.558 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:24.558 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:24.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:24.558 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:24.559 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:24.559 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:24.559 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:24.559 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:24.560 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:24.560 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:24.560 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:24.560 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:24.561 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:24.561 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:24.561 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:24.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:24.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:24.562 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:24.562 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:24.562 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:24.562 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:24.563 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:24.563 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:24.563 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:24.563 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:24.564 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:24.564 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:24.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:24.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:24.565 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:24.565 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:24.565 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:24.565 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:47:24.566 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:24.566 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:24.566 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:24.566 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:47:24.567 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:24.567 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:24.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:24.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:24.568 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:24.568 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:24.568 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:24.568 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:24.569 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:24.569 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:24.569 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:24.570 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:24.570 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:24.570 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:24.570 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:24.571 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:24.571 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:24.571 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:24.571 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:24.572 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:24.572 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:24.572 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:24.573 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.486035_153.1 2021-09-06T11:47:29.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:29.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:29.576 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:29.625 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:29.625 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:29.961 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:29.962 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:29.962 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:29.963 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:29.963 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:29.963 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:29.963 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:29.964 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:29.964 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:29.964 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:29.964 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:29.965 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:29.965 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:29.965 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:29.965 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:29.965 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:29.966 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:29.966 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:29.966 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:29.966 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:29.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:29.967 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:29.967 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:29.967 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:29.967 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:29.968 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:29.968 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:29.968 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:29.968 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:29.968 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:29.969 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:29.969 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:29.969 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:29.969 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:29.969 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:29.970 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:29.970 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:29.970 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:47:29.970 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:29.971 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:29.971 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:29.971 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:47:29.971 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:29.971 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:29.972 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:29.972 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:29.972 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:29.972 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:29.972 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:29.973 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:29.973 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:29.973 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:29.973 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:29.974 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:29.974 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:29.974 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:29.974 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:29.975 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:29.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:29.975 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:29.975 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:29.975 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:29.976 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:29.976 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:29.977 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.486035_153.1 2021-09-06T11:47:34.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:34.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:34.980 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:35.027 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:35.028 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:35.100 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:35.100 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:35.108 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:35.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:35.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:35.108 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:47:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:47:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:35.115 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:35.115 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:35.115 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:35.116 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.484822_152.1 2021-09-06T11:47:40.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:40.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:40.119 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:40.168 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:40.168 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:40.243 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:40.243 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:40.243 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:40.244 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:40.244 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:40.244 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:40.244 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:40.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:40.245 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:40.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:40.245 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:40.245 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:40.246 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:40.246 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:40.246 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:40.246 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:40.246 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:40.247 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:40.247 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:40.247 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:40.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:40.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:40.248 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:40.248 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:40.248 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:40.249 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:40.249 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:40.249 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:40.249 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:40.249 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:40.250 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:40.250 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:40.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:40.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:40.251 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:40.251 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:40.251 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:40.252 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:47:40.252 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:40.252 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:40.252 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:40.253 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:47:40.253 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:40.253 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:40.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:40.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:40.254 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:40.254 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:40.254 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:40.255 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:40.255 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:40.255 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:40.255 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:40.256 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:40.256 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:40.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:40.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:40.256 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:40.257 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:40.257 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:40.257 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:40.257 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:40.258 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:40.258 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:40.259 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.508454_157.1 2021-09-06T11:47:45.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:45.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:45.263 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:45.356 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:45.356 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:45.730 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:45.731 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:45.731 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:45.731 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:45.732 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:45.732 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:45.732 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:45.732 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:45.733 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:45.733 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:45.733 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:45.733 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:45.734 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:45.734 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:45.734 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:45.734 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:45.735 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:45.735 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:45.735 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:45.735 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:45.736 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:45.736 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:45.736 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:45.736 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:45.736 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:45.737 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:45.737 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:45.737 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:45.737 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:45.737 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:45.738 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:45.738 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:45.738 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:45.738 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:45.739 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:45.739 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:45.739 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:45.739 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.611335_165.1", 2021-09-06T11:47:45.740 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:45.740 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:45.740 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:45.740 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.611335Z", 2021-09-06T11:47:45.740 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:45.741 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:45.741 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:45.741 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:45.741 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:45.742 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:45.742 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:45.742 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:45.742 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:45.742 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:45.743 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:45.743 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:45.743 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:45.743 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:45.743 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:45.744 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:45.744 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:45.744 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:45.744 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:45.745 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:45.745 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:45.745 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:45.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.611335_165.1 2021-09-06T11:47:50.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:50.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:50.749 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:50.832 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:50.832 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:50.901 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:50.901 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:50.901 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:50.902 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:50.902 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:50.902 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:50.902 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:50.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:50.903 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:50.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:50.903 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:50.904 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:50.904 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:50.904 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:50.904 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:50.904 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:50.905 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:50.905 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:50.905 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:50.905 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:50.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:50.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:50.906 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:50.906 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:50.906 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:50.907 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:50.907 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:50.907 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:50.907 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:50.908 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:50.908 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:50.908 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:50.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:50.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:50.909 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:50.909 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:50.909 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:50.909 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:47:50.910 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:50.910 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:50.910 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:50.910 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:47:50.911 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:50.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:50.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:50.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:50.912 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:50.912 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:50.912 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:50.912 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:50.912 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:50.913 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:50.913 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:50.913 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:50.913 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:50.914 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:50.914 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:50.914 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:50.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:50.914 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:50.915 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:50.915 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:50.915 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:50.915 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:50.917 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.508454_157.1 2021-09-06T11:47:55.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:55.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:47:55.919 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:47:55.969 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:47:55.969 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:47:56.041 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:47:56.041 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:47:56.042 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:47:56.042 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:47:56.042 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:47:56.042 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:47:56.043 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:56.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:56.043 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:47:56.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:56.043 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:47:56.044 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:56.044 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:56.044 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:47:56.044 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:56.045 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:47:56.045 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:56.045 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:47:56.045 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:56.045 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:56.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:56.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:56.046 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:47:56.046 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:56.047 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:56.047 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:47:56.047 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:56.047 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:56.047 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:56.048 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:47:56.048 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:56.048 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:56.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:56.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:56.049 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:47:56.049 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:56.049 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:56.049 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:47:56.049 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:56.050 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:56.050 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:56.050 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:47:56.050 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:56.051 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:56.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:56.051 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:47:56.051 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:47:56.052 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:47:56.052 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:47:56.052 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:47:56.052 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:47:56.053 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:47:56.053 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:47:56.053 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:47:56.053 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:47:56.054 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:56.054 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:56.054 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:47:56.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:47:56.055 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:47:56.055 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:47:56.055 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:47:56.055 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:47:56.055 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:47:56.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.544174_161.1 2021-09-06T11:48:01.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:01.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:01.059 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:01.102 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:01.103 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:01.171 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:01.171 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:01.172 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:01.173 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:01.174 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:01.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:01.176 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:01.177 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:01.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:01.179 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:48:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:01.180 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:48:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:01.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:01.182 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:01.183 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:01.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:01.185 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:01.185 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:01.187 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.508454_157.1 2021-09-06T11:48:06.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:06.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:06.190 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:06.271 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:06.271 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:06.421 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:06.421 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:06.421 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:06.422 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:06.422 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:06.422 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:06.422 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:06.423 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:06.423 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:06.423 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:06.423 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:06.424 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:06.424 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:06.424 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:06.424 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:06.425 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:06.425 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:06.425 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:06.425 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:06.425 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:06.426 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:06.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:06.426 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:06.426 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:06.427 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:06.427 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:06.427 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:06.427 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:06.428 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:06.428 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:06.428 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:06.428 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:06.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:06.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:06.429 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:06.429 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:06.429 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:06.430 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:48:06.430 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:06.430 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:06.430 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:06.430 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:48:06.431 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:06.431 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:06.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:06.431 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:06.431 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:06.432 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:06.432 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:06.432 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:06.432 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:06.432 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:06.433 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:06.433 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:06.433 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:06.433 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:06.433 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:06.434 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:06.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:06.434 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:06.434 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:06.435 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:06.435 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:06.435 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:06.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.484822_152.1 2021-09-06T11:48:11.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:11.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:11.438 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:11.526 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:11.526 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:11.655 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:11.655 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:11.656 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:11.656 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:11.656 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:11.656 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:11.656 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:11.657 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:11.657 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:11.657 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:11.657 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:11.658 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:11.658 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:11.658 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:11.658 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:11.659 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:11.659 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:11.659 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:11.659 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:11.659 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:11.660 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:11.660 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:11.660 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:11.660 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:11.660 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:11.661 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:11.661 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:11.661 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:11.661 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:11.662 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:11.662 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:11.662 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:11.662 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:11.662 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:11.663 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:11.663 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:11.663 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:11.663 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:48:11.664 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:11.664 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:11.664 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:11.664 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:48:11.664 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:11.665 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:11.665 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:11.665 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:11.665 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:11.666 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:11.666 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:11.666 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:11.666 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:11.667 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:11.667 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:11.667 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:11.667 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:11.668 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:11.668 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:11.668 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:11.668 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:11.668 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:11.669 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:11.669 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:11.669 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:11.669 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:11.670 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.738114_171.1 2021-09-06T11:48:16.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:16.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:16.673 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:16.752 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:16.752 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:17.299 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:17.299 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:17.300 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:17.300 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:17.300 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:17.300 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:17.300 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:17.301 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:17.301 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:17.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:17.301 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:17.301 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:17.302 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:17.302 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:17.302 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:17.302 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:17.303 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:17.303 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:17.303 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:17.303 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:17.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:17.304 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:17.304 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:17.304 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:17.304 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:17.304 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:17.305 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:17.305 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:17.305 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:17.305 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:17.306 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:17.306 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:17.306 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:17.306 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:17.306 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:17.307 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:17.307 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:17.307 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:48:17.307 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:17.307 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:17.308 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:17.308 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:48:17.308 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:17.308 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:17.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:17.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:17.309 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:17.309 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:17.309 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:17.310 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:17.310 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:17.310 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:17.310 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:17.311 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:17.311 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:17.311 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:17.311 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:17.312 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:17.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:17.312 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:17.312 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:17.312 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:17.313 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:17.313 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:17.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.486035_153.1 2021-09-06T11:48:22.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:22.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:22.316 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:22.363 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:22.363 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:22.436 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:22.436 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:22.436 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:22.437 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:22.437 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:22.437 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:22.437 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:22.437 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:22.438 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:22.438 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:22.438 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:22.438 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:22.439 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:22.439 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:22.439 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:22.439 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:22.440 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:22.440 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:22.440 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:22.440 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:22.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:22.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:22.441 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:22.441 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:22.441 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:22.442 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:22.442 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:22.442 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:22.442 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:22.443 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:22.443 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:22.443 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:22.443 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:22.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:22.444 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:22.444 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:22.444 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:22.444 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:48:22.445 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:22.445 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:22.445 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:22.445 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:48:22.445 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:22.446 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:22.446 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:22.446 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:22.446 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:22.447 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:22.447 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:22.447 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:22.447 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:22.447 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:22.448 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:22.448 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:22.448 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:22.448 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:22.448 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:22.449 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:22.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:22.449 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:22.449 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:22.449 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:22.450 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:22.450 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:22.451 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.484822_152.1 2021-09-06T11:48:27.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:27.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:27.454 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:27.496 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:27.496 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:27.562 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:27.562 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:27.562 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:27.563 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:27.563 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:27.563 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:27.563 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:27.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:27.564 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:27.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:27.564 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:27.564 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:27.564 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:27.565 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:27.565 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:27.565 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:27.565 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:27.566 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:27.566 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:27.566 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:27.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:27.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:27.567 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:27.567 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:27.567 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:27.567 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:27.567 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:27.568 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:27.568 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:27.568 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:27.568 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:27.568 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:27.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:27.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:27.569 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:27.569 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:27.569 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:27.570 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:48:27.570 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:27.570 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:27.570 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:27.571 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:48:27.571 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:27.571 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:27.571 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:27.571 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:27.572 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:27.572 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:27.572 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:27.572 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:27.572 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:27.573 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:27.573 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:27.573 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:27.573 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:27.573 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:27.574 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:27.574 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:27.574 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:27.574 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:27.574 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:27.575 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:27.575 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:27.575 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:27.576 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.503662_155.1 2021-09-06T11:48:32.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:32.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:32.579 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:32.646 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:32.647 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:33.038 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:33.038 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:33.038 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:33.039 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:33.039 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:33.039 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:33.039 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:33.040 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:33.040 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:33.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:33.040 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:33.040 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:33.041 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:33.041 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:33.041 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:33.041 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:33.041 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:33.042 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:33.042 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:33.042 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:33.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:33.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:33.043 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:33.043 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:33.043 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:33.043 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:33.044 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:33.044 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:33.044 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:33.044 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:33.044 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:33.045 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:33.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:33.045 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:33.045 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:33.045 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:33.046 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:33.046 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:48:33.046 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:33.046 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:33.046 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:33.047 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:48:33.047 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:33.047 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:33.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:33.047 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:33.048 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:33.048 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:33.048 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:33.048 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:33.049 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:33.049 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:33.049 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:33.049 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:33.049 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:33.050 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:33.050 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:33.050 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:33.050 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:33.051 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:33.051 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:33.051 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:33.051 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:33.051 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:33.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.484822_152.1 2021-09-06T11:48:38.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:38.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:38.054 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:38.130 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:38.131 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:38.281 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:38.282 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:38.282 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:38.282 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:38.282 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:38.283 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:38.283 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:38.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:38.283 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:38.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:38.284 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:38.284 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:38.284 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:38.284 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:38.285 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:38.285 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:38.285 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:38.285 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:38.286 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:38.286 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:38.286 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:38.286 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:38.286 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:38.287 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:38.287 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:38.287 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:38.287 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:38.288 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:38.288 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:38.288 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:38.288 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:38.289 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:38.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:38.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:38.289 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:38.289 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:38.290 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:38.290 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:48:38.290 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:38.290 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:38.291 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:38.291 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:48:38.291 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:38.291 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:38.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:38.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:38.292 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:38.292 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:38.293 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:38.293 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:38.293 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:38.293 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:38.293 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:38.294 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:38.294 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:38.294 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:38.294 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:38.295 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:38.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:38.295 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:38.295 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:38.295 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:38.296 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:38.296 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:38.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.486035_153.1 2021-09-06T11:48:43.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:43.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:43.299 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:43.343 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:43.343 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:43.439 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:43.440 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:43.440 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:43.440 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:43.440 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:43.440 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:43.441 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:43.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:43.441 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:43.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:43.442 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:43.442 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:43.442 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:43.442 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:43.442 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:43.443 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:43.443 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:43.443 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:43.443 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:43.443 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:43.444 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:43.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:43.444 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:43.444 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:43.444 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:43.445 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:43.445 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:43.445 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:43.445 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:43.446 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:43.446 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:43.446 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:43.446 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:43.446 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:43.447 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:43.447 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:43.447 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:43.447 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:48:43.447 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:43.448 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:43.448 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:43.448 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:48:43.448 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:43.448 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:43.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:43.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:43.449 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:43.449 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:43.449 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:43.450 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:43.450 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:43.450 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:43.450 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:43.450 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:43.451 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:43.451 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:43.451 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:43.451 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:43.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:43.452 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:43.452 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:43.452 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:43.452 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:43.452 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:43.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.508454_157.1 2021-09-06T11:48:48.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:48.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:48.456 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:48.497 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:48.498 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:48.566 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:48.566 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:48.567 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:48.567 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:48.567 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:48.567 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:48.568 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:48.568 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:48.568 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:48.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:48.568 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:48.569 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:48.569 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:48.569 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:48.569 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:48.569 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:48.570 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:48.570 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:48.570 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:48.570 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:48.571 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:48.571 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:48.571 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:48.571 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:48.571 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:48.571 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:48.572 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:48.572 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:48.572 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:48.572 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:48.573 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:48.573 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:48.573 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:48.573 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:48.573 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:48.573 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:48.574 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:48.574 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T11:48:48.574 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:48.574 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:48.575 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:48.575 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T11:48:48.575 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:48.575 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:48.575 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:48.575 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:48.576 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:48.576 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:48.576 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:48.576 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:48.577 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:48.577 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:48.577 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:48.577 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:48.577 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:48.578 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:48.578 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:48.578 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:48.578 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:48.579 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:48.579 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:48.579 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:48.579 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:48.579 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:48.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.489232_154.1 2021-09-06T11:48:53.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:53.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:53.583 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:53.665 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:53.666 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:53.882 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:53.882 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:53.883 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:53.883 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:53.883 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:53.883 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:53.884 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:53.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:53.884 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:53.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:53.885 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:53.885 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:53.885 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:53.885 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:53.886 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:53.886 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:53.886 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:53.886 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:53.887 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:53.887 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:53.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:53.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:53.888 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:53.888 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:53.888 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:53.888 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:53.889 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:53.889 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:53.889 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:53.889 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:53.889 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:53.890 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:53.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:53.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:53.890 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:53.891 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:53.891 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:53.891 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:48:53.891 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:53.892 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:53.892 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:53.892 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:48:53.892 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:53.893 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:53.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:53.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:53.893 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:53.893 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:53.894 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:53.894 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:53.894 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:53.894 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:53.895 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:53.895 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:53.895 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:53.895 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:53.896 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:53.896 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:53.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:53.896 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:53.896 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:53.897 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:53.897 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:53.897 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:53.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.503662_155.1 2021-09-06T11:48:58.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:58.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:48:58.902 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:48:58.948 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:48:58.949 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:48:59.021 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:48:59.021 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:48:59.021 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:48:59.022 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:48:59.022 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:48:59.022 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:48:59.022 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:59.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:59.023 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:48:59.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:59.023 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:48:59.024 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:59.024 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:59.024 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:48:59.024 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:59.025 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:48:59.025 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:59.025 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:48:59.025 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:59.026 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:59.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:59.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:59.026 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:48:59.027 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:59.027 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:59.027 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:48:59.027 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:59.028 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:59.028 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:59.028 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:48:59.028 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:59.029 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:59.029 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:59.029 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:59.029 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:48:59.030 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:59.030 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:59.030 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.565140_164.1", 2021-09-06T11:48:59.030 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:59.031 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:59.031 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:59.031 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.565140Z", 2021-09-06T11:48:59.031 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:59.031 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:59.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:59.032 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:48:59.032 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:48:59.033 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:48:59.033 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:48:59.033 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:48:59.033 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:48:59.034 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:48:59.034 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:48:59.034 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:48:59.034 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:48:59.034 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:59.035 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:59.035 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:48:59.035 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:48:59.036 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:48:59.036 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:48:59.036 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:48:59.036 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:48:59.036 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:48:59.038 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.565140_164.1 2021-09-06T11:49:04.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:04.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:04.040 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:04.119 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:04.120 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:04.667 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:04.667 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:04.667 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:04.668 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:04.668 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:04.668 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:04.668 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:04.668 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:04.669 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:04.669 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:04.669 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:04.669 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:04.670 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:04.670 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:04.670 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:04.670 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:04.670 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:04.671 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:04.671 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:04.671 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:04.671 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:04.672 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:04.672 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:04.672 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:04.672 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:04.672 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:04.673 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:04.673 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:04.673 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:04.673 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:04.673 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:04.674 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:04.674 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:04.674 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:04.674 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:04.675 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:04.675 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:04.675 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T11:49:04.675 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:04.676 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:04.676 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:04.676 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T11:49:04.676 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:04.676 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:04.677 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:04.677 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:04.677 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:04.677 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:04.677 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:04.678 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:04.678 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:04.678 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:04.679 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:04.679 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:04.679 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:04.679 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:04.679 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:04.680 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:04.680 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:04.680 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:04.680 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:04.681 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:04.681 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:04.681 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:04.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.489232_154.1 2021-09-06T11:49:09.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:09.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:09.685 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:09.733 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:09.733 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:09.808 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:09.808 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:09.809 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:09.809 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:09.809 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:09.809 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:09.810 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:09.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:09.810 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:09.810 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:09.811 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:09.811 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:09.811 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:09.811 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:09.811 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:09.812 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:09.812 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:09.812 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:09.812 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:09.813 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:09.813 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:09.813 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:09.813 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:09.813 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:09.814 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:09.814 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:09.814 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:09.814 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:09.815 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:09.815 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:09.815 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:09.815 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:09.816 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:09.816 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:09.816 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:09.816 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:09.816 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:09.817 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.539819_159.1", 2021-09-06T11:49:09.817 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:09.817 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:09.817 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:09.818 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.539819Z", 2021-09-06T11:49:09.818 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:09.818 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:09.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:09.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:09.819 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:09.819 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:09.819 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:09.820 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:09.820 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:09.820 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:09.820 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:09.820 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:09.821 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:09.821 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:09.821 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:09.821 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:09.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:09.822 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:09.822 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:09.822 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:09.823 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:09.823 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:09.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.539819_159.1 2021-09-06T11:49:14.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:14.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:14.828 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:14.870 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:14.870 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:14.934 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:14.934 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:14.934 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:14.935 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:14.935 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:14.935 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:14.935 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:14.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:14.936 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:14.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:14.936 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:14.937 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:14.937 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:14.937 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:14.937 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:14.937 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:14.938 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:14.938 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:14.938 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:14.938 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:14.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:14.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:14.939 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:14.939 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:14.939 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:14.940 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:14.940 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:14.940 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:14.940 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:14.941 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:14.941 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:14.941 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:14.941 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:14.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:14.942 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:14.942 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:14.942 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:14.943 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:49:14.943 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:14.943 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:14.943 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:14.944 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:49:14.944 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:14.944 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:14.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:14.944 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:14.945 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:14.945 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:14.945 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:14.945 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:14.946 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:14.946 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:14.946 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:14.946 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:14.947 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:14.947 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:14.947 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:14.947 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:14.948 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:14.948 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:14.948 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:14.948 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:14.949 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:14.949 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:14.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.503662_155.1 2021-09-06T11:49:19.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:19.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:19.953 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:20.019 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:20.020 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:20.520 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:20.520 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:20.520 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:20.520 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:20.521 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:20.521 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:20.521 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:20.521 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:20.522 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:20.522 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:20.522 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:20.522 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:20.523 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:20.523 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:20.523 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:20.523 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:20.523 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:20.524 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:20.524 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:20.524 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:20.524 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:20.524 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:20.525 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:20.525 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:20.525 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:20.525 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:20.526 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:20.526 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:20.526 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:20.526 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:20.526 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:20.527 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:20.527 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:20.527 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:20.527 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:20.527 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:20.528 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:20.528 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T11:49:20.528 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:20.528 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:20.528 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:20.529 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T11:49:20.529 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:20.529 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:20.529 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:20.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:20.530 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:20.530 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:20.530 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:20.530 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:20.531 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:20.531 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:20.531 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:20.531 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:20.532 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:20.532 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:20.532 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:20.532 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:20.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:20.533 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:20.533 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:20.533 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:20.533 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:20.534 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:20.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.489232_154.1 2021-09-06T11:49:25.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:25.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:25.538 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:25.619 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:25.619 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:26.061 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:26.061 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:26.062 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:26.062 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:26.062 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:26.062 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:26.063 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:26.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:26.063 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:26.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:26.064 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:26.064 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:26.064 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:26.064 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:26.065 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:26.065 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:26.065 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:26.065 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:26.066 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:26.066 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:26.066 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:26.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:26.066 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:26.067 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:26.067 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:26.067 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:26.067 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:26.067 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:26.068 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:26.068 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:26.068 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:26.068 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:26.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:26.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:26.069 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:26.069 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:26.069 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:26.070 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:49:26.070 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:26.070 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:26.070 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:26.071 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:49:26.071 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:26.071 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:26.071 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:26.072 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:26.072 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:26.072 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:26.072 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:26.072 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:26.073 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:26.073 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:26.073 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:26.073 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:26.074 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:26.074 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:26.074 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:26.074 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:26.075 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:26.075 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:26.075 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:26.075 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:26.075 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:26.076 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:26.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.486035_153.1 2021-09-06T11:49:31.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:31.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:31.079 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:31.128 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:31.128 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:31.405 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:31.406 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:31.406 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:31.406 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:31.406 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:31.407 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:31.407 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:31.407 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:31.407 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:31.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:31.408 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:31.408 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:31.408 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:31.409 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:31.409 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:31.409 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:31.409 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:31.410 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:31.410 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:31.410 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:31.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:31.411 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:31.411 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:31.411 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:31.412 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:31.412 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:31.412 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:31.412 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:31.413 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:31.413 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:31.413 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:31.413 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:31.414 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:31.414 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:31.414 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:31.414 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:31.415 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:31.415 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:49:31.415 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:31.415 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:31.416 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:31.416 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:49:31.416 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:31.416 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:31.417 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:31.417 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:31.417 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:31.417 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:31.418 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:31.418 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:31.418 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:31.419 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:31.419 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:31.419 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:31.419 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:31.420 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:31.420 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:31.420 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:31.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:31.421 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:31.421 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:31.421 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:31.421 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:31.422 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:31.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.484822_152.1 2021-09-06T11:49:36.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:36.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:36.425 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:36.472 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:36.473 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:36.548 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:36.548 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:36.548 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:36.549 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:36.549 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:36.549 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:36.549 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:36.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:36.550 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:36.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:36.550 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:36.550 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:36.551 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:36.551 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:36.551 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:36.551 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:36.551 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:36.552 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:36.552 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:36.552 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:36.552 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:36.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:36.553 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:36.553 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:36.553 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:36.553 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:36.554 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:36.554 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:36.554 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:36.554 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:36.554 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:36.555 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:36.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:36.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:36.555 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:36.556 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:36.556 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:36.556 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:49:36.556 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:36.556 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:36.557 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:36.557 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:49:36.557 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:36.557 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:36.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:36.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:36.558 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:36.558 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:36.558 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:36.559 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:36.559 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:36.559 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:36.559 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:36.559 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:36.560 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:36.560 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:36.560 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:36.560 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:36.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:36.561 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:36.561 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:36.561 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:36.561 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:36.561 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:36.562 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.508454_157.1 2021-09-06T11:49:41.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:41.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:41.565 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:41.627 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:41.627 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:41.700 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:41.700 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:41.700 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:41.701 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:41.701 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:41.701 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:41.701 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:41.702 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:41.702 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:41.702 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:41.702 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:41.703 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:41.703 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:41.703 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:41.703 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:41.704 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:41.704 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:41.704 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:41.704 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:41.705 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:41.705 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:41.705 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:41.705 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:41.706 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:41.706 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:41.706 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:41.706 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:41.707 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:41.707 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:41.707 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:41.707 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:41.708 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:41.708 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:41.708 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:41.708 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:41.709 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:41.709 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:41.709 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:49:41.709 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:41.710 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:41.710 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:41.710 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:49:41.710 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:41.710 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:41.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:41.711 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:41.711 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:41.711 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:41.712 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:41.712 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:41.712 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:41.712 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:41.713 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:41.713 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:41.713 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:41.713 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:41.714 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:41.714 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:41.714 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:41.714 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:41.715 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:41.715 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:41.715 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:41.715 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:41.717 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.508454_157.1 2021-09-06T11:49:46.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:46.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:46.720 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:46.779 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:46.779 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:47.019 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:47.019 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:47.019 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:47.019 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:47.019 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:47.020 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:47.020 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:47.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:47.020 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:47.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:47.021 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:47.021 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:47.021 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:47.021 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:47.022 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:47.022 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:47.022 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:47.022 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:47.023 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:47.023 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:47.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:47.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:47.023 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:47.024 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:47.024 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:47.024 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:47.024 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:47.024 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:47.025 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:47.025 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:47.025 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:47.025 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:47.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:47.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:47.026 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:47.026 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:47.026 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:47.026 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:49:47.027 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:47.027 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:47.027 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:47.027 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:49:47.027 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:47.028 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:47.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:47.028 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:47.028 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:47.028 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:47.029 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:47.029 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:47.029 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:47.029 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:47.029 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:47.030 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:47.030 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:47.030 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:47.030 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:47.030 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:47.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:47.031 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:47.031 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:47.031 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:47.031 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:47.032 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:47.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.486035_153.1 2021-09-06T11:49:52.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:52.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:52.034 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:52.115 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:52.116 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:52.237 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:52.238 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:52.238 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:52.238 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:52.239 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:52.239 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:52.239 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:52.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:52.240 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:52.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:52.240 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:52.241 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:52.241 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:52.241 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:52.241 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:52.241 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:52.242 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:52.242 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:52.242 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:52.242 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:52.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:52.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:52.243 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:52.243 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:52.244 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:52.244 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:52.244 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:52.244 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:52.244 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:52.245 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:52.245 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:52.245 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:52.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:52.246 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:52.246 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:52.246 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:52.246 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:52.246 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:49:52.247 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:52.247 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:52.247 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:52.247 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:49:52.247 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:52.248 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:52.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:52.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:52.248 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:52.249 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:52.249 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:52.249 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:52.249 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:52.249 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:52.250 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:52.250 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:52.250 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:52.250 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:52.250 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:52.251 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:52.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:52.251 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:52.251 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:52.251 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:52.252 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:52.252 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:52.253 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.511634_158.1 2021-09-06T11:49:57.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:57.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:49:57.256 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:49:57.306 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:49:57.306 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:49:57.719 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:49:57.719 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:49:57.720 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:49:57.720 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:49:57.720 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:49:57.720 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:49:57.721 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:57.721 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:57.721 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:49:57.721 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:57.721 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:49:57.722 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:57.722 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:57.722 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:49:57.722 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:57.723 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:49:57.723 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:57.723 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:49:57.723 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:57.723 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:57.724 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:57.724 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:57.724 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:49:57.724 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:57.724 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:57.725 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:49:57.725 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:57.725 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:57.725 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:57.726 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:49:57.726 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:57.726 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:57.726 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:57.726 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:57.727 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:49:57.727 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:57.727 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:57.727 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.484822_152.1", 2021-09-06T11:49:57.728 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:57.728 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:57.728 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:57.728 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.484822Z", 2021-09-06T11:49:57.728 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:57.729 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:57.729 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:57.729 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:49:57.729 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:49:57.730 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:49:57.730 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:49:57.730 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:49:57.730 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:49:57.730 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:49:57.731 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:49:57.731 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:49:57.731 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:49:57.731 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:57.732 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:57.732 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:49:57.732 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:49:57.732 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:49:57.732 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:49:57.733 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:49:57.733 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:49:57.733 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:49:57.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.484822_152.1 2021-09-06T11:50:02.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:02.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:02.737 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:02.782 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:02.783 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:02.852 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:02.852 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:02.853 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:02.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:02.855 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:02.856 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:02.857 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:02.858 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:02.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:02.860 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:02.861 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:02.862 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:02.863 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:02.864 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:02.865 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.802344_180.1", 2021-09-06T11:50:02.866 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:02.867 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.802344Z", 2021-09-06T11:50:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:02.868 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:02.869 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:02.870 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:02.871 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:02.872 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:02.873 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:02.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:02.875 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:02.876 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:02.876 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:02.878 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.802344_180.1 2021-09-06T11:50:07.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:07.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:07.882 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:07.943 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:07.943 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:08.046 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:08.046 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:08.047 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:08.047 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:08.047 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:08.047 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:08.048 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:08.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:08.048 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:08.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:08.048 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:08.049 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:08.049 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:08.049 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:08.049 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:08.049 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:08.050 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:08.050 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:08.050 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:08.050 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:08.050 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:08.051 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:08.051 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:08.051 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:08.051 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:08.051 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:08.052 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:08.052 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:08.052 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:08.052 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:08.052 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:08.053 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:08.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:08.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:08.053 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:08.054 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:08.054 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:08.054 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:50:08.054 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:08.054 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:08.055 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:08.055 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:50:08.055 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:08.055 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:08.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:08.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:08.056 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:08.056 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:08.056 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:08.056 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:08.057 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:08.057 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:08.057 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:08.057 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:08.057 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:08.058 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:08.058 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:08.058 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:08.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:08.058 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:08.059 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:08.059 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:08.059 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:08.059 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:08.060 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.486035_153.1 2021-09-06T11:50:13.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:13.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:13.063 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:13.118 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:13.119 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:13.376 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:13.377 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:13.377 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:13.377 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:13.377 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:13.378 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:13.378 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:13.378 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:13.378 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:13.378 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:13.379 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:13.379 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:13.379 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:13.379 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:13.379 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:13.380 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:13.380 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:13.380 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:13.380 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:13.380 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:13.381 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:13.381 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:13.381 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:13.381 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:13.382 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:13.382 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:13.382 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:13.382 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:13.382 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:13.383 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:13.383 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:13.383 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:13.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:13.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:13.384 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:13.384 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:13.384 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:13.384 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:50:13.384 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:13.385 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:13.385 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:13.385 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:50:13.385 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:13.386 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:13.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:13.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:13.386 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:13.386 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:13.387 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:13.387 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:13.387 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:13.387 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:13.387 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:13.388 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:13.388 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:13.388 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:13.388 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:13.389 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:13.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:13.389 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:13.389 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:13.389 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:13.390 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:13.390 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:13.390 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.503662_155.1 2021-09-06T11:50:18.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:18.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:18.393 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:18.484 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:18.484 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:18.784 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:18.785 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:18.785 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:18.785 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:18.785 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:18.786 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:18.786 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:18.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:18.787 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:18.787 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:18.787 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:18.787 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:18.788 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:18.788 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:18.788 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:18.788 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:18.789 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:18.789 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:18.789 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:18.789 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:18.790 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:18.790 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:18.790 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:18.790 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:18.791 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:18.791 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:18.791 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:18.791 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:18.792 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:18.792 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:18.792 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:18.792 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:18.793 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:18.793 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:18.793 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:18.793 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:18.794 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:18.794 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:50:18.794 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:18.794 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:18.795 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:18.795 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:50:18.795 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:18.795 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:18.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:18.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:18.796 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:18.796 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:18.797 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:18.797 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:18.797 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:18.797 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:18.797 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:18.798 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:18.798 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:18.798 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:18.798 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:18.799 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:18.799 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:18.799 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:18.799 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:18.800 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:18.800 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:18.800 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:18.801 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.486035_153.1 2021-09-06T11:50:23.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:23.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:23.804 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:23.892 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:23.893 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:24.381 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:24.382 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:24.382 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:24.382 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:24.383 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:24.383 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:24.383 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:24.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:24.384 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:24.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:24.384 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:24.384 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:24.384 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:24.385 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:24.385 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:24.385 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:24.385 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:24.386 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:24.386 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:24.386 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:24.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:24.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:24.387 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:24.387 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:24.387 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:24.388 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:24.388 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:24.388 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:24.388 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:24.389 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:24.389 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:24.389 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:24.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:24.390 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:24.390 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:24.390 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:24.390 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:24.391 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:50:24.391 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:24.391 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:24.391 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:24.392 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:50:24.392 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:24.392 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:24.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:24.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:24.393 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:24.393 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:24.393 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:24.393 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:24.394 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:24.394 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:24.394 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:24.394 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:24.395 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:24.395 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:24.395 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:24.395 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:24.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:24.396 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:24.396 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:24.396 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:24.396 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:24.397 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:24.397 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.544174_161.1 2021-09-06T11:50:29.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:29.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:29.399 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:29.442 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:29.442 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:29.514 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:29.515 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:29.515 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:29.515 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:29.516 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:29.516 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:29.516 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:29.516 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:29.516 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:29.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:29.522 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:29.522 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:50:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:29.524 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:29.524 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:29.524 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:50:29.524 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:29.524 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:29.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:29.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:29.525 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:29.525 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:29.526 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:29.526 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:29.526 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:29.526 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:29.527 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:29.527 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:29.527 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:29.527 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:29.527 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:29.529 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:29.529 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:29.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.544174_161.1 2021-09-06T11:50:34.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:34.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:34.533 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:34.575 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:34.576 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:34.640 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:34.641 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:34.641 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:34.641 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:34.642 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:34.642 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:34.642 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:34.643 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:34.645 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:34.645 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:34.646 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:34.646 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:34.646 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:34.647 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:34.647 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:34.647 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:34.647 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:34.648 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:34.648 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:34.648 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:34.649 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:34.649 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:34.649 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:34.650 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:34.650 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:34.650 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:34.651 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:34.651 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:34.651 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:34.652 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:34.652 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:34.652 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:34.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:34.653 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:34.653 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:34.653 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:34.654 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:34.654 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:50:34.654 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:34.655 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:34.655 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:34.655 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:50:34.656 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:34.656 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:34.656 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:34.657 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:34.657 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:34.657 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:34.658 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:34.658 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:34.658 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:34.659 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:34.659 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:34.659 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:34.660 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:34.660 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:34.660 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:34.661 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:34.661 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:34.661 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:34.662 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:34.662 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:34.662 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:34.663 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:34.665 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.544174_161.1 2021-09-06T11:50:39.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:39.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:39.668 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:39.733 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:39.734 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:39.806 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:39.806 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:39.807 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:39.807 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:39.807 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:39.807 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:39.808 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:39.808 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:39.808 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:39.808 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:39.808 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:39.809 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:39.809 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:39.809 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:39.809 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:39.810 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:39.810 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:39.810 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:39.810 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:39.811 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:39.811 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:39.811 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:39.811 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:39.811 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:39.812 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:39.812 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:39.812 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:39.813 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:39.813 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:39.813 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:39.813 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:39.813 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:39.814 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:39.814 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:39.814 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:39.814 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:39.815 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:39.815 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.561874_163.1", 2021-09-06T11:50:39.815 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:39.815 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:39.815 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:39.816 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.561874Z", 2021-09-06T11:50:39.816 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:39.816 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:39.816 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:39.817 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:39.817 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:39.817 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:39.817 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:39.818 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:39.818 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:39.818 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:39.818 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:39.818 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:39.819 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:39.819 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:39.819 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:39.819 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:39.820 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:39.820 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:39.820 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:39.820 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:39.820 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:39.821 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:39.822 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.561874_163.1 2021-09-06T11:50:44.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:44.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:44.825 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:44.878 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:44.879 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:44.950 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:44.951 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:44.951 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:44.951 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:44.952 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:44.952 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:44.952 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:44.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:44.953 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:44.953 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:44.953 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:44.953 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:44.954 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:44.954 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:44.954 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:44.955 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:44.955 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:44.955 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:44.956 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:44.956 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:44.956 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:44.957 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:44.958 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:44.958 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:44.958 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:44.958 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:44.959 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:44.959 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:44.959 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:44.959 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:44.960 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:44.960 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:44.960 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:44.960 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:44.961 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:50:44.961 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:44.961 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:44.961 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:44.961 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:50:44.962 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:44.962 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:44.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:44.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:44.963 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:44.963 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:44.963 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:44.963 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:44.964 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:44.964 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:44.964 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:44.964 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:44.965 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:44.965 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:44.965 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:44.965 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:44.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:44.966 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:44.966 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:44.966 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:44.966 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:44.967 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:44.968 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.544174_161.1 2021-09-06T11:50:49.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:49.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:49.971 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:50.019 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:50.019 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:50.185 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:50.185 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:50.185 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:50.186 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:50.186 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:50.186 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:50.186 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:50.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:50.187 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:50.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:50.187 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:50.188 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:50.188 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:50.188 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:50.188 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:50.189 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:50.189 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:50.189 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:50.189 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:50.190 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:50.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:50.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:50.190 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:50.191 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:50.191 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:50.191 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:50.191 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:50.192 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:50.192 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:50.192 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:50.192 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:50.193 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:50.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:50.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:50.193 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:50.194 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:50.194 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:50.194 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:50:50.194 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:50:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:50.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:50.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:50.196 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:50.196 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:50.197 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:50.197 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:50.197 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:50.197 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:50.198 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:50.198 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:50.198 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:50.198 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:50.199 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:50.199 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:50.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:50.199 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:50.200 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:50.200 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:50.200 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:50.200 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:50.201 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.544174_161.1 2021-09-06T11:50:55.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:55.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:50:55.204 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:50:55.246 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:50:55.246 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:50:55.311 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:50:55.312 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:50:55.312 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:50:55.312 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:50:55.313 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:50:55.313 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:50:55.313 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:55.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:55.313 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:50:55.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:55.314 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:50:55.314 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:55.314 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:55.315 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:50:55.315 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:55.315 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:50:55.315 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:55.315 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:50:55.316 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:55.316 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:55.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:55.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:55.317 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:50:55.317 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:55.317 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:55.317 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:50:55.317 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:55.318 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:55.318 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:55.318 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:50:55.318 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:55.319 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:55.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:55.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:55.319 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:50:55.319 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:55.320 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:55.320 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T11:50:55.320 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:55.320 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:55.321 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:55.321 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T11:50:55.321 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:55.321 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:55.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:55.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:50:55.322 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:50:55.322 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:50:55.322 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:50:55.322 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:50:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:50:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:50:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:50:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:50:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:50:55.324 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:55.324 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:55.324 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:50:55.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:50:55.324 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:50:55.325 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:50:55.325 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:50:55.325 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:50:55.325 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:50:55.326 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630928741.670299_257.1 target=1_1630928091.489232_154.1 2021-09-06T11:51:00.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2021-09-06T11:51:00.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:51:00.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:51:00.331 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T11:51:00.412 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:51:00.412 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:51:00.532 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:51:00.529+0000 7fc79d329380 0 failed reading obj info from .rgw.root:zone_info.7556a536-76a8-4c42-adbe-379e64c0e1a8: (2) No such file or directory 2021-09-06T11:51:00.533 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:51:00.529+0000 7fc79d329380 0 WARNING: could not read zone params for zone id=7556a536-76a8-4c42-adbe-379e64c0e1a8 name=test-zone1 2021-09-06T11:51:00.533 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:51:00.532+0000 7fc79d329380 0 failed reading obj info from .rgw.root:zone_info.c248f771-5d9b-48a2-a793-2bc85cc1f251: (2) No such file or directory 2021-09-06T11:51:00.533 INFO:teuthology.orchestra.run.smithi139.stderr:2021-09-06T11:51:00.532+0000 7fc79d329380 0 WARNING: could not read zone params for zone id=c248f771-5d9b-48a2-a793-2bc85cc1f251 name=test-zone3 2021-09-06T11:51:00.572 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:51:00.573 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:51:00.573 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone2", 2021-09-06T11:51:00.573 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-06T11:51:00.573 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-06T11:51:00.574 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-06T11:51:00.574 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-06T11:51:00.574 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-06T11:51:00.574 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-06T11:51:00.575 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-06T11:51:00.575 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-06T11:51:00.575 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-06T11:51:00.575 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-06T11:51:00.575 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-06T11:51:00.576 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-06T11:51:00.576 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-06T11:51:00.576 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-06T11:51:00.577 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2021-09-06T11:51:00.577 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "1XCNWZEYEDFPNNWL", 2021-09-06T11:51:00.577 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "s07ZRdFO0iVWGMNXYGQHTrCafM0bZd9C" 2021-09-06T11:51:00.577 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:00.577 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2021-09-06T11:51:00.578 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:00.578 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2021-09-06T11:51:00.578 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:51:00.578 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-06T11:51:00.579 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2021-09-06T11:51:00.579 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2021-09-06T11:51:00.579 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-06T11:51:00.579 INFO:teuthology.orchestra.run.smithi139.stdout: "compression_type": "random" 2021-09-06T11:51:00.580 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:51:00.580 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:00.580 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-06T11:51:00.580 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0 2021-09-06T11:51:00.580 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:51:00.581 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:51:00.581 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:51:00.581 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:51:00.581 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-06T11:51:00.581 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:51:00.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:51:00.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T11:51:00.584 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T11:51:00.667 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:51:00.668 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:51:00.773 INFO:teuthology.orchestra.run.smithi139.stderr:Sending period to new master zone 7556a536-76a8-4c42-adbe-379e64c0e1a8 2021-09-06T11:51:01.835 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:51:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:51:01.835 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 5, 2021-09-06T11:51:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "predecessor_uuid": "8245bc5e-afe7-418b-afcf-7fd599759612", 2021-09-06T11:51:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": [], 2021-09-06T11:51:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "period_map": { 2021-09-06T11:51:01.836 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:51:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "zonegroups": [ 2021-09-06T11:51:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "test-zonegroup", 2021-09-06T11:51:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zonegroup", 2021-09-06T11:51:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: "api_name": "", 2021-09-06T11:51:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: "is_master": "true", 2021-09-06T11:51:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:51:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:51:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:51:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames": [], 2021-09-06T11:51:01.839 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames_s3website": [], 2021-09-06T11:51:01.839 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:51:01.839 INFO:teuthology.orchestra.run.smithi139.stdout: "zones": [ 2021-09-06T11:51:01.839 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:01.840 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:51:01.840 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone2", 2021-09-06T11:51:01.840 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:51:01.840 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2021-09-06T11:51:01.840 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:51:01.841 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2021-09-06T11:51:01.841 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "true", 2021-09-06T11:51:01.841 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:51:01.841 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2021-09-06T11:51:01.841 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2021-09-06T11:51:01.842 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "false", 2021-09-06T11:51:01.842 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2021-09-06T11:51:01.842 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2021-09-06T11:51:01.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:01.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:01.843 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "4487d356-c18b-4713-bd68-02dfa183d02a", 2021-09-06T11:51:01.843 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone4", 2021-09-06T11:51:01.843 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:51:01.843 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2021-09-06T11:51:01.843 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:51:01.844 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2021-09-06T11:51:01.844 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "true", 2021-09-06T11:51:01.844 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:51:01.844 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2021-09-06T11:51:01.844 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "pubsub", 2021-09-06T11:51:01.845 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "false", 2021-09-06T11:51:01.845 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [ 2021-09-06T11:51:01.845 INFO:teuthology.orchestra.run.smithi139.stdout: "test-zone1" 2021-09-06T11:51:01.845 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:51:01.846 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2021-09-06T11:51:01.846 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:01.846 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:01.846 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:51:01.846 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone1", 2021-09-06T11:51:01.847 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:51:01.847 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T11:51:01.847 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:51:01.847 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2021-09-06T11:51:01.848 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "true", 2021-09-06T11:51:01.848 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:51:01.848 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2021-09-06T11:51:01.848 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2021-09-06T11:51:01.848 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "true", 2021-09-06T11:51:01.849 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2021-09-06T11:51:01.849 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2021-09-06T11:51:01.849 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:01.849 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:01.849 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T11:51:01.850 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-zone3", 2021-09-06T11:51:01.850 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2021-09-06T11:51:01.850 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi094.front.sepia.ceph.com:8001" 2021-09-06T11:51:01.850 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:51:01.851 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": "false", 2021-09-06T11:51:01.851 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": "true", 2021-09-06T11:51:01.851 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2021-09-06T11:51:01.851 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": "false", 2021-09-06T11:51:01.851 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2021-09-06T11:51:01.852 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": "true", 2021-09-06T11:51:01.852 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2021-09-06T11:51:01.852 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "" 2021-09-06T11:51:01.852 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:51:01.852 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:51:01.853 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_targets": [ 2021-09-06T11:51:01.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:01.853 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default-placement", 2021-09-06T11:51:01.853 INFO:teuthology.orchestra.run.smithi139.stdout: "tags": [], 2021-09-06T11:51:01.853 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": [ 2021-09-06T11:51:01.854 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD" 2021-09-06T11:51:01.854 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:51:01.854 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:51:01.854 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:51:01.854 INFO:teuthology.orchestra.run.smithi139.stdout: "default_placement": "default-placement", 2021-09-06T11:51:01.855 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:51:01.855 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_policy": { 2021-09-06T11:51:01.855 INFO:teuthology.orchestra.run.smithi139.stdout: "groups": [] 2021-09-06T11:51:01.855 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:51:01.856 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:51:01.856 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2021-09-06T11:51:01.856 INFO:teuthology.orchestra.run.smithi139.stdout: "short_zone_ids": [ 2021-09-06T11:51:01.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:01.860 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T11:51:01.861 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 4134315483 2021-09-06T11:51:01.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:01.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:01.862 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "4487d356-c18b-4713-bd68-02dfa183d02a", 2021-09-06T11:51:01.862 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 2859952350 2021-09-06T11:51:01.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:01.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:01.863 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:51:01.863 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 772393743 2021-09-06T11:51:01.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:01.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:51:01.864 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T11:51:01.864 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 242578107 2021-09-06T11:51:01.864 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:51:01.864 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:51:01.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:01.865 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T11:51:01.865 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T11:51:01.865 INFO:teuthology.orchestra.run.smithi139.stdout: "period_config": { 2021-09-06T11:51:01.865 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_quota": { 2021-09-06T11:51:01.866 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2021-09-06T11:51:01.866 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2021-09-06T11:51:01.866 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2021-09-06T11:51:01.866 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2021-09-06T11:51:01.867 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2021-09-06T11:51:01.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:01.867 INFO:teuthology.orchestra.run.smithi139.stdout: "user_quota": { 2021-09-06T11:51:01.867 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2021-09-06T11:51:01.867 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2021-09-06T11:51:01.868 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2021-09-06T11:51:01.868 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2021-09-06T11:51:01.868 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2021-09-06T11:51:01.868 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:51:01.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:51:01.869 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T11:51:01.869 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_name": "test-realm", 2021-09-06T11:51:01.869 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:51:01.869 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:51:01.870 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-09-06T11:52:01.872 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-45 2021-09-06T11:52:01.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:01.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:01.949 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:02.173 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:02.174 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:02.224 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:52:02.240 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:52:02.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:52:02.241 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:02.241 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:52:02.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:52:02.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:52:02.242 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:52:02.242 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:52:02.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:52:02.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:52:02.243 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630929121.899863_275.1", 2021-09-06T11:52:02.243 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:52:01.899863Z" 2021-09-06T11:52:02.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:52:02.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:52:02.244 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:02.244 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:52:02.244 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:52:02.244 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:52:02.246 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:52:02.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:02.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:02.248 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:02.295 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:02.296 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:02.375 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:02.375 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:02.376 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:02.376 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:02.376 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:02.376 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:02.377 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:02.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:02.377 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:02.377 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:02.378 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:02.378 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:02.378 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:02.378 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:02.379 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:02.379 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:02.379 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:02.379 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:02.380 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:02.380 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:02.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:02.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:02.380 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:02.381 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:02.381 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:02.381 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:02.381 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:02.382 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:02.382 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:02.382 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:02.382 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:02.383 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:02.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:02.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:02.383 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:02.384 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:02.384 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:02.384 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:02.384 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:02.385 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:02.385 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:02.385 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:02.385 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:02.385 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:02.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:02.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:02.386 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:02.386 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:02.387 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:02.387 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:02.387 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:02.387 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:02.388 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:02.388 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:02.388 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:02.388 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:02.388 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:02.389 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:02.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:02.389 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:02.389 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:02.389 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:02.390 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:02.390 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:02.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:07.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:07.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:07.394 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:07.443 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:07.444 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:07.522 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:07.522 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:07.523 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:07.523 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:07.523 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:07.523 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:07.524 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:07.524 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:07.524 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:07.524 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:07.525 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:07.525 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:07.525 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:07.525 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:07.526 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:07.526 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:07.526 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:07.526 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:07.527 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:07.527 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:07.527 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:07.527 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:07.528 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:07.528 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:07.528 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:07.528 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:07.528 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:07.529 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:07.529 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:07.529 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:07.529 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:07.530 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:07.530 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:07.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:07.530 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:07.530 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:07.531 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:07.531 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:07.531 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:07.531 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:07.532 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:07.532 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:07.532 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:07.532 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:07.533 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:07.533 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:07.533 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:07.533 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:07.533 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:07.534 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:07.534 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:07.534 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:07.534 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:07.535 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:07.535 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:07.535 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:07.535 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:07.536 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:07.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:07.536 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:07.536 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:07.537 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:07.537 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:07.537 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:07.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:12.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:12.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:12.541 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:12.590 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:12.591 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:12.662 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:12.663 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:12.663 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:12.663 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:12.664 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:12.664 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:12.664 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:12.664 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:12.665 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:12.665 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:12.665 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:12.665 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:12.665 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:12.666 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:12.666 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:12.666 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:12.666 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:12.667 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:12.667 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:12.667 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:12.667 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:12.667 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:12.668 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:12.668 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:12.668 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:12.669 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:12.669 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:12.669 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:12.669 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:12.669 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:12.670 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:12.670 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:12.670 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:12.670 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:12.671 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:12.671 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:12.671 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:12.671 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:12.671 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:12.672 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:12.672 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:12.672 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:12.672 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:12.673 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:12.673 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:12.673 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:12.673 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:12.673 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:12.674 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:12.674 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:12.674 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:12.674 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:12.675 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:12.675 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:12.675 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:12.675 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:12.675 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:12.676 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:12.676 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:12.676 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:12.676 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:12.677 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:12.677 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:12.677 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:12.678 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:17.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:17.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:17.681 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:17.729 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:17.730 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:17.810 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:17.811 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:17.811 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:17.811 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:17.811 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:17.812 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:17.812 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:17.812 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:17.812 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:17.812 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:17.813 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:17.813 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:17.813 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:17.813 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:17.814 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:17.814 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:17.814 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:17.814 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:17.815 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:17.815 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:17.815 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:17.815 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:17.816 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:17.816 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:17.816 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:17.816 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:17.817 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:17.817 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:17.817 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:17.817 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:17.818 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:17.818 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:17.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:17.818 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:17.819 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:17.819 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:17.819 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:17.819 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:17.820 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:17.820 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:17.820 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:17.820 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:17.821 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:17.821 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:17.821 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:17.821 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:17.822 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:17.822 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:17.822 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:17.822 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:17.823 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:17.823 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:17.823 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:17.823 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:17.824 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:17.824 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:17.824 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:17.824 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:17.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:17.825 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:17.825 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:17.825 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:17.825 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:17.826 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:17.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:22.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:22.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:22.829 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:22.878 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:22.878 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:22.950 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:22.950 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:22.950 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:22.951 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:22.951 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:22.951 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:22.951 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:22.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:22.952 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:22.952 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:22.952 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:22.953 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:22.953 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:22.953 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:22.953 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:22.954 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:22.954 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:22.954 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:22.954 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:22.955 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:22.955 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:22.955 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:22.956 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:22.956 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:22.957 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:22.957 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:22.957 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:22.957 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:22.957 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:22.958 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:22.958 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:22.958 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:22.959 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:22.959 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:22.959 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:22.959 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:22.960 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:22.960 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:22.960 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:22.961 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:22.961 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:22.961 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:22.961 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:22.962 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:22.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:22.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:22.962 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:22.963 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:22.963 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:22.963 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:22.964 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:22.964 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:22.964 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:22.965 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:22.965 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:22.966 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:22.966 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:22.966 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:22.966 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:22.967 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:22.967 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:22.967 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:22.968 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:22.968 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:22.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:27.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:27.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:27.973 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:28.021 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:28.022 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:28.095 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:28.096 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:28.096 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:28.096 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:28.097 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:28.097 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:28.097 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:28.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:28.097 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:28.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:28.098 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:28.098 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:28.098 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:28.099 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:28.099 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:28.099 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:28.099 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:28.100 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:28.100 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:28.100 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:28.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:28.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:28.101 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:28.101 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:28.101 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:28.101 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:28.101 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:28.102 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:28.102 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:28.102 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:28.102 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:28.103 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:28.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:28.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:28.103 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:28.103 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:28.104 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:28.104 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:28.104 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:28.104 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:28.104 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:28.105 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:28.105 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:28.105 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:28.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:28.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:28.106 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:28.106 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:28.106 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:28.107 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:28.107 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:28.107 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:28.107 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:28.107 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:28.108 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:28.108 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:28.108 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:28.108 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:28.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:28.109 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:28.109 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:28.109 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:28.109 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:28.110 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:28.111 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:31.940 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T11:52:31.939+0000 7fc4c2a2c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:52:31.943 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2021-09-06T11:52:31.943+0000 7f1434690700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T11:52:33.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:33.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:33.113 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:33.164 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:33.164 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:33.250 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:33.251 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:33.251 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:33.251 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:33.251 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:33.252 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:33.252 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:33.252 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:33.252 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:33.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:33.253 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:33.253 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:33.253 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:33.253 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:33.254 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:33.254 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:33.254 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:33.254 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:33.255 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:33.255 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:33.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:33.255 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:33.256 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:33.256 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:33.256 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:33.256 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:33.257 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:33.257 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:33.257 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:33.257 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:33.257 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:33.258 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:33.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:33.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:33.258 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:33.259 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:33.259 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:33.259 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:33.259 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:33.260 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:33.260 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:33.260 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:33.260 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:33.260 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:33.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:33.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:33.261 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:33.261 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:33.262 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:33.262 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:33.262 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:33.262 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:33.263 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:33.263 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:33.263 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:33.263 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:33.263 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:33.264 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:33.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:33.264 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:33.264 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:33.265 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:33.265 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:33.265 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:33.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:38.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:38.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:38.270 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:38.318 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:38.318 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:38.394 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:38.394 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:38.394 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:38.394 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:38.395 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:38.395 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:38.395 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:38.396 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:38.396 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:38.396 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:38.396 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:38.397 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:38.397 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:38.397 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:38.397 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:38.398 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:38.398 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:38.398 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:38.398 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:38.399 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:38.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:38.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:38.399 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:38.399 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:38.400 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:38.400 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:38.400 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:38.400 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:38.401 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:38.401 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:38.401 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:38.401 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:38.401 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:38.402 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:38.402 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:38.402 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:38.403 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:38.403 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:38.403 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:38.403 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:38.403 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:38.404 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:38.404 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:38.404 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:38.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:38.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:38.405 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:38.405 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:38.405 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:38.406 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:38.406 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:38.406 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:38.407 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:38.407 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:38.407 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:38.407 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:38.408 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:38.408 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:38.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:38.409 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:38.409 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:38.409 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:38.409 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:38.409 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:38.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:43.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:43.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:43.414 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:43.461 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:43.462 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:43.533 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:43.533 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:43.534 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:43.534 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:43.534 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:43.534 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:43.535 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:43.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:43.535 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:43.535 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:43.536 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:43.536 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:43.536 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:43.536 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:43.537 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:43.537 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:43.537 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:43.537 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:43.538 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:43.538 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:43.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:43.538 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:43.539 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:43.539 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:43.539 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:43.539 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:43.539 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:43.540 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:43.540 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:43.540 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:43.540 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:43.541 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:43.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:43.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:43.541 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:43.542 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:43.542 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:43.542 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:43.542 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:43.542 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:43.543 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:43.543 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:43.543 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:43.543 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:43.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:43.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:43.544 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:43.544 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:43.544 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:43.545 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:43.545 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:43.545 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:43.545 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:43.546 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:43.546 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:43.546 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:43.546 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:43.547 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:43.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:43.547 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:43.547 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:43.547 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:43.548 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:43.548 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:43.549 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:48.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:48.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:48.552 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:48.601 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:48.601 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:48.681 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:48.681 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:48.681 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:48.682 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:48.682 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:48.682 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:48.682 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:48.683 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:48.683 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:48.683 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:48.683 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:48.684 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:48.684 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:48.684 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:48.684 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:48.685 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:48.685 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:48.685 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:48.685 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:48.686 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:48.686 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:48.686 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:48.686 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:48.687 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:48.687 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:48.687 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:48.688 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:48.688 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:48.688 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:48.688 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:48.688 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:48.689 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:48.689 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:48.689 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:48.689 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:48.690 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:48.690 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:48.690 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:48.690 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:48.691 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:48.691 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:48.691 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:48.691 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:48.691 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:48.692 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:48.692 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:48.692 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:48.692 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:48.693 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:48.693 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:48.693 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:48.693 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:48.694 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:48.694 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:48.694 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:48.694 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:48.695 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:48.695 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:48.695 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:48.695 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:48.696 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:48.696 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:48.696 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:48.696 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:48.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:53.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:53.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:53.700 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:53.749 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:53.749 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:53.824 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:53.825 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:53.825 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:53.825 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:53.825 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:53.826 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:53.826 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:53.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:53.826 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:53.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:53.827 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:53.827 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:53.827 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:53.828 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:53.828 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:53.828 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:53.828 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:53.829 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:53.829 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:53.829 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:53.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:53.830 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:53.830 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:53.830 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:53.830 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:53.831 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:53.831 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:53.831 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:53.831 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:53.831 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:53.832 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:53.832 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:53.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:53.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:53.832 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:53.833 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:53.833 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:53.833 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:53.833 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:53.834 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:53.834 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:53.834 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:53.834 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:53.834 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:53.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:53.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:53.835 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:53.835 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:53.836 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:53.836 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:53.836 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:53.836 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:53.837 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:53.837 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:53.837 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:53.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:53.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:53.838 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:53.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:53.838 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:53.838 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:53.839 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:53.839 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:53.839 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:53.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:52:58.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:58.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:52:58.843 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:52:58.891 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:52:58.891 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:52:58.965 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:52:58.965 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:52:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:52:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:52:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:52:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:52:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:52:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:52:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:52:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:52:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:52:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:58.970 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:58.970 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:58.970 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:58.970 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:52:58.971 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:58.971 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:58.971 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:52:58.971 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:58.971 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:58.972 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:58.972 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:52:58.972 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:58.972 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:58.973 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:58.973 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:58.973 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:52:58.973 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:58.973 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:58.974 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.486035_153.1", 2021-09-06T11:52:58.974 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:58.974 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:58.974 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:58.975 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.486035Z", 2021-09-06T11:52:58.975 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:58.975 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:58.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:58.975 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:52:58.976 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:52:58.976 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:52:58.976 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:52:58.976 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:52:58.976 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:52:58.977 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:52:58.977 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:52:58.977 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:52:58.977 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:52:58.977 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:58.978 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:58.978 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:52:58.978 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:52:58.978 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:52:58.978 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:52:58.979 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:52:58.979 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:52:58.979 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:52:58.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.486035_153.1 2021-09-06T11:53:03.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:03.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:03.982 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:04.024 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:04.024 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:04.089 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:04.090 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:04.090 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:04.090 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:04.091 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:04.091 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:04.091 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:04.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:04.092 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:04.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:04.092 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:04.092 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:04.092 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:04.093 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:04.093 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:04.093 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:04.093 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:04.094 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:04.094 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:04.094 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:04.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:04.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:04.095 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:04.096 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:04.096 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:04.096 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:04.097 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:04.097 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:04.097 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:04.097 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:04.098 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:04.098 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:04.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:04.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:04.099 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:04.099 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:04.099 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:04.099 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:53:04.099 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:04.100 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:04.100 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:04.100 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:53:04.100 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:04.101 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:04.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:04.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:04.101 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:04.102 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:04.102 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:04.102 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:04.102 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:04.103 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:04.103 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:04.103 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:04.103 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:04.104 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:04.104 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:04.104 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:04.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:04.105 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:04.105 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:04.105 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:04.106 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:04.106 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:04.107 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.508454_157.1 2021-09-06T11:53:09.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:09.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:09.111 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:09.161 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:09.161 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:09.393 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:09.394 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:09.394 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:09.394 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:09.395 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:09.395 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:09.395 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:09.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:09.395 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:09.396 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:09.396 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:09.396 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:09.396 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:09.397 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:09.397 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:09.397 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:09.397 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:09.397 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:09.398 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:09.398 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:09.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:09.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:09.399 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:09.399 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:09.399 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:09.399 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:09.399 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:09.400 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:09.400 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:09.400 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:09.400 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:09.401 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:09.401 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:09.401 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:09.401 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:09.401 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:09.402 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:09.402 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:53:09.402 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:09.402 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:09.403 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:09.403 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:53:09.403 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:09.403 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:09.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:09.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:09.404 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:09.404 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:09.404 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:09.404 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:09.405 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:09.405 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:09.405 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:09.405 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:09.405 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:09.406 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:09.406 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:09.406 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:09.406 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:09.406 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:09.407 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:09.407 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:09.407 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:09.407 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:09.408 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.544174_161.1 2021-09-06T11:53:14.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:14.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:14.412 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:14.491 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:14.491 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:14.597 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:14.597 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:14.597 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:14.597 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:14.598 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:14.598 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:14.598 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:14.598 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:14.599 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:14.599 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:14.599 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:14.599 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:14.600 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:14.600 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:14.600 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:14.600 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:14.603 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:14.603 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:14.603 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:14.603 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:14.603 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:14.604 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:14.604 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:14.604 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:14.604 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:14.605 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:14.605 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:14.605 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:14.605 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:14.606 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.996301_187.1", 2021-09-06T11:53:14.606 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:14.606 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:14.606 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.996301Z", 2021-09-06T11:53:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:14.610 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:14.610 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:14.610 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:14.610 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:14.611 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:14.611 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:14.611 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:14.611 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:14.611 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:14.612 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:14.613 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.996301_187.1 2021-09-06T11:53:19.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:19.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:19.616 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:19.660 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:19.661 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:19.736 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:19.736 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:19.736 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:19.737 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:19.737 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:19.737 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:19.737 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:19.738 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:19.738 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:19.738 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:19.738 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:19.739 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:19.739 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:19.739 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:19.739 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:19.739 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:19.740 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:19.740 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:19.740 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:19.740 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:19.741 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:19.741 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:19.741 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:19.741 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:19.742 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:19.742 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:19.742 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:19.742 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:19.742 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:19.743 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:19.743 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:19.743 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:19.743 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:19.744 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:19.744 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:19.744 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:19.744 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:19.745 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.741261_172.1", 2021-09-06T11:53:19.745 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:19.745 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:19.745 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:19.746 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.741261Z", 2021-09-06T11:53:19.746 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:19.746 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:19.746 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:19.747 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:19.747 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:19.747 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:19.747 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:19.748 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:19.748 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:19.748 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:19.748 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:19.748 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:19.749 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:19.749 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:19.749 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:19.750 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:19.750 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:19.750 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:19.751 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:19.751 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:19.751 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:19.751 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:19.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.741261_172.1 2021-09-06T11:53:24.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:24.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:24.756 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:24.799 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:24.800 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:24.875 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:24.875 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:24.875 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:24.876 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:24.876 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:24.876 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:24.876 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:24.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:24.877 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:24.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:24.877 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:24.877 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:24.878 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:24.878 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:24.878 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:24.878 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:24.879 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:24.879 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:24.879 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:24.879 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:24.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:24.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:24.880 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:24.880 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:24.880 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:24.881 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:24.881 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:24.881 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:24.881 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:24.882 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:24.882 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:24.882 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:24.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:24.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:24.883 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:24.883 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:24.883 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:24.883 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:53:24.884 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:24.884 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:24.884 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:24.884 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:53:24.885 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:24.885 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:24.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:24.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:24.886 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:24.886 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:24.886 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:24.886 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:24.887 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:24.887 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:24.887 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:24.887 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:24.888 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:24.888 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:24.888 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:24.888 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:24.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:24.889 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:24.889 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:24.889 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:24.889 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:24.890 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:24.891 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.738114_171.1 2021-09-06T11:53:29.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:29.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:29.893 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:29.936 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:29.937 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:30.002 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:30.002 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:30.003 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:30.004 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:30.005 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:30.006 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:30.007 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:53:30.010 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:53:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:30.011 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:30.012 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:30.013 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:30.014 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:30.015 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:30.016 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:30.016 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:30.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.508454_157.1 2021-09-06T11:53:35.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:35.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:35.020 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:35.063 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:35.063 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:35.232 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:35.232 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:35.232 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:35.232 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:53:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:53:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:35.246 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:35.246 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:35.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:35.246 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:35.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.544174_161.1 2021-09-06T11:53:40.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:40.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:40.249 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:40.339 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:40.339 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:40.763 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:40.764 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:40.764 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:40.764 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:40.764 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:40.765 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:40.765 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:40.765 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:40.765 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:40.766 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:40.766 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:40.766 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:40.766 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:40.767 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:40.767 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:40.767 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:40.767 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:40.767 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:40.768 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:40.768 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:40.768 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:40.768 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:40.769 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:40.769 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:40.769 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:40.769 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:40.770 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:40.770 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:40.770 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:40.770 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:40.771 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:40.771 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:40.771 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:40.771 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:40.772 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:40.772 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:40.772 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:40.772 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:53:40.773 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:40.773 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:40.773 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:40.773 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:53:40.773 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:40.774 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:40.774 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:40.774 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:40.774 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:40.775 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:40.775 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:40.775 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:40.775 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:40.776 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:40.776 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:40.776 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:40.776 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:40.777 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:40.777 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:40.777 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:40.777 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:40.778 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:40.778 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:40.778 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:40.778 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:40.778 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:40.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.508454_157.1 2021-09-06T11:53:45.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:45.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:45.783 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:45.860 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:45.860 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:46.466 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:46.466 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:46.467 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:46.468 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:46.469 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:46.470 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:46.471 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:46.472 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:46.473 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:46.474 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.676443_168.1", 2021-09-06T11:53:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:46.475 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.676443Z", 2021-09-06T11:53:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:46.476 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:46.477 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:46.478 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:46.479 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:46.480 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:46.481 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:46.481 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:46.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.676443_168.1 2021-09-06T11:53:51.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:51.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:51.485 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:51.535 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:51.535 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:51.880 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:51.883 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:51.883 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:51.883 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:51.883 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.670393_167.1", 2021-09-06T11:53:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.670393Z", 2021-09-06T11:53:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:51.894 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:51.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:51.894 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:51.895 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.670393_167.1 2021-09-06T11:53:56.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:56.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:53:56.898 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:53:56.947 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:53:56.947 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:53:57.019 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:53:57.020 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:53:57.020 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:53:57.020 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:53:57.020 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:53:57.021 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:53:57.021 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:57.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:57.021 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:53:57.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:57.022 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:53:57.022 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:57.022 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:57.023 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:53:57.023 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:57.023 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:53:57.023 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:57.023 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:53:57.024 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:57.024 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:57.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:57.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:57.025 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:53:57.025 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:57.025 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:57.025 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:53:57.026 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:57.026 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:57.026 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:57.026 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:53:57.026 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:57.027 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:57.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:57.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:57.027 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:53:57.027 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:57.028 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:57.028 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.561874_163.1", 2021-09-06T11:53:57.028 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:57.028 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:57.029 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:57.029 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.561874Z", 2021-09-06T11:53:57.029 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:57.029 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:57.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:57.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:53:57.030 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:53:57.030 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:53:57.030 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:53:57.031 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:53:57.031 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:53:57.031 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:53:57.031 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:53:57.031 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:53:57.032 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:53:57.032 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:57.032 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:57.032 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:53:57.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:53:57.033 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:53:57.033 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:53:57.033 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:53:57.033 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:53:57.034 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:53:57.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.561874_163.1 2021-09-06T11:54:02.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:02.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:02.038 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:02.081 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:02.081 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:02.157 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:02.157 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:02.158 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:02.158 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:02.158 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:02.158 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:02.159 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:02.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:02.159 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:02.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:02.159 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:02.160 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:02.160 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:02.160 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:02.160 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:02.161 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:02.161 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:02.161 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:02.161 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:02.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:02.162 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:02.162 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:02.162 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:02.162 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:02.163 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:02.163 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:02.163 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:02.163 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:02.164 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:02.164 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:02.164 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:02.164 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:02.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:02.165 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:02.165 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:02.165 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:02.165 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:02.166 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T11:54:02.166 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:02.166 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:02.166 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:02.167 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T11:54:02.167 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:02.167 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:02.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:02.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:02.168 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:02.168 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:02.168 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:02.168 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:02.169 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:02.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:02.171 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:02.171 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:02.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.489232_154.1 2021-09-06T11:54:07.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:07.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:07.176 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:07.267 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:07.267 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:07.681 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:07.681 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:07.682 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:07.682 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:07.682 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:07.682 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:07.683 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:07.683 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:07.683 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:07.683 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:07.683 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:07.684 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:07.684 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:07.684 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:07.684 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:07.684 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:07.685 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:07.685 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:07.685 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:07.685 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:07.685 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:07.686 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:07.686 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:07.686 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:07.686 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:07.687 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:07.687 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:07.687 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:07.687 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:07.687 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:07.688 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:07.688 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:07.688 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:07.688 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:07.689 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:07.689 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:07.689 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:07.689 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:54:07.690 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:07.690 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:07.690 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:07.690 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:54:07.691 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:07.691 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:07.691 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:07.691 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:07.692 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:07.692 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:07.692 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:07.692 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:07.692 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:07.693 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:07.693 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:07.693 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:07.693 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:07.694 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:07.694 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:07.694 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:07.694 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:07.695 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:07.695 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:07.695 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:07.695 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:07.695 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:07.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T11:54:12.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:12.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:12.699 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:12.750 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:12.750 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:12.822 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:12.822 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:12.823 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:12.823 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:12.823 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:12.823 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:12.823 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:12.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:12.824 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:12.824 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:12.824 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:12.825 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:12.825 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:12.825 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:12.825 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:12.826 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:12.826 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:12.826 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:12.826 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:12.827 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:12.827 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:12.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:12.827 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:12.827 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:12.828 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:12.828 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:12.828 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:12.828 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:12.829 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:12.829 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:12.829 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:12.829 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:12.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:12.830 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:12.830 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:12.830 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:12.830 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:12.831 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:54:12.831 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:12.831 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:12.831 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:12.832 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:54:12.832 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:12.832 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:12.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:12.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:12.833 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:12.833 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:12.833 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:12.833 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:12.834 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:12.834 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:12.834 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:12.834 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:12.834 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:12.835 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:12.835 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:12.835 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:12.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:12.836 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:12.836 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:12.836 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:12.836 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:12.837 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:12.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.508454_157.1 2021-09-06T11:54:17.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:17.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:17.840 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:17.918 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:17.919 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:18.340 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:18.341 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:18.341 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:18.341 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:18.341 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:18.342 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:18.342 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:18.342 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:18.342 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:18.342 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:18.343 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:18.343 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:18.343 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:18.343 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:18.343 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:18.344 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:18.344 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:18.344 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:18.344 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:18.344 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:18.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:18.345 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:18.345 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:18.345 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:18.345 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:18.346 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:18.346 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:18.346 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:18.346 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:18.347 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:18.347 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:18.347 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:18.347 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:18.347 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:18.348 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:18.348 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:18.348 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:18.348 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:54:18.348 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:18.349 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:18.349 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:18.349 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:54:18.349 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:18.350 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:18.350 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:18.350 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:18.350 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:18.350 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:18.351 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:18.351 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:18.351 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:18.351 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:18.352 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:18.352 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:18.352 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:18.352 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:18.352 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:18.353 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:18.353 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:18.353 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:18.353 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:18.353 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:18.354 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:18.354 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:18.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.508454_157.1 2021-09-06T11:54:23.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:23.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:23.358 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:23.406 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:23.406 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:23.502 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:23.503 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:23.503 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:23.503 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:23.504 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:23.504 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:23.504 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:23.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:23.504 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:23.505 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:23.505 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:23.505 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:23.505 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:23.505 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:23.506 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:23.506 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:23.506 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:23.506 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:23.506 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:23.507 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:23.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:23.507 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:23.507 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:23.508 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:23.508 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:23.508 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:23.508 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:23.508 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:23.509 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:23.509 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:23.509 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:23.509 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:23.509 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:23.509 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:23.510 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:23.510 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:23.510 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:23.510 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:54:23.511 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:23.511 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:23.511 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:23.511 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:54:23.511 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:23.512 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:23.512 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:23.512 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:23.512 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:23.512 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:23.513 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:23.513 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:23.513 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:23.513 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:23.514 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:23.514 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:23.514 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:23.514 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:23.514 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:23.515 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:23.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:23.515 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:23.515 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:23.515 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:23.516 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:23.516 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:23.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T11:54:28.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:28.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:28.519 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:28.567 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:28.567 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:28.643 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:28.643 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:28.643 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:28.644 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:28.644 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:28.644 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:28.644 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:28.645 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:28.645 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:28.645 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:28.645 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:28.645 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:28.646 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:28.646 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:28.646 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:28.646 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:28.647 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:28.647 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:28.647 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:28.647 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:28.647 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:28.648 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:28.648 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:28.648 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:28.648 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:28.649 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:28.649 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:28.649 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:28.649 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:28.649 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:28.650 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:28.650 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:28.650 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:28.650 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:28.651 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:28.651 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:28.651 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:28.651 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.670393_167.1", 2021-09-06T11:54:28.652 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:28.652 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:28.652 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:28.652 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.670393Z", 2021-09-06T11:54:28.653 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:28.653 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:28.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:28.653 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:28.653 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:28.654 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:28.654 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:28.654 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:28.654 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:28.654 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:28.655 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:28.655 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:28.655 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:28.656 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:28.656 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:28.656 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:28.656 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:28.656 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:28.657 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:28.657 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:28.657 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:28.657 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:28.659 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.670393_167.1 2021-09-06T11:54:33.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:33.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:33.662 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:33.759 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:33.759 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:33.946 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:33.946 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:33.947 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:33.947 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:33.947 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:33.947 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:33.948 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:33.948 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:33.948 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:33.948 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:33.948 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:33.949 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:33.949 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:33.949 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:33.949 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:33.950 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:33.950 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:33.950 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:33.950 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:33.951 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:33.951 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:33.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:33.951 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:33.951 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:33.952 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:33.952 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:33.952 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:33.952 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:33.953 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:33.953 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:33.953 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:33.953 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:33.954 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:33.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:33.954 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:33.954 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:33.954 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:33.955 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T11:54:33.955 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:33.955 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:33.955 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:33.956 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T11:54:33.956 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:33.956 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:33.956 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:33.956 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:33.957 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:33.957 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:33.957 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:33.957 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:33.958 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:33.958 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:33.958 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:33.958 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:33.959 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:33.959 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:33.959 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:33.959 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:33.960 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:33.960 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:33.960 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:33.960 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:33.960 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:33.961 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:33.962 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.489232_154.1 2021-09-06T11:54:38.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:38.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:38.965 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:39.045 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:39.045 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:39.205 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:39.206 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:39.206 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:39.206 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:39.207 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:39.207 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:39.207 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:39.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:39.208 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:39.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:39.208 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:39.208 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:39.209 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:39.209 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:39.209 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:39.209 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:39.209 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:39.210 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:39.210 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:39.210 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:39.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:39.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:39.211 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:39.211 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:39.211 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:39.212 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:39.212 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:39.212 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:39.212 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:39.212 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:39.213 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:39.213 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:39.213 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:39.213 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:39.214 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:39.214 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:39.214 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:39.214 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:54:39.214 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:39.215 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:39.215 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:39.215 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:54:39.216 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:39.216 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:39.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:39.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:39.216 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:39.217 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:39.217 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:39.217 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:39.217 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:39.217 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:39.218 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:39.218 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:39.218 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:39.218 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:39.219 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:39.219 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:39.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:39.219 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:39.220 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:39.220 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:39.220 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:39.220 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:39.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.738114_171.1 2021-09-06T11:54:44.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:44.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:44.225 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:44.318 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:44.319 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:44.822 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.793863_177.1", 2021-09-06T11:54:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.793863Z", 2021-09-06T11:54:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:44.835 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:44.835 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:44.835 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:44.835 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:44.835 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:44.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:44.837 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:44.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.793863_177.1 2021-09-06T11:54:49.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:49.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:49.841 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:49.889 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:49.890 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:49.967 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:49.968 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:49.968 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:49.968 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:49.968 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:49.968 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:49.969 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:49.969 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:49.969 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:49.969 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:49.970 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:49.970 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:49.970 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:49.970 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:49.971 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:49.971 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:49.971 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:49.971 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:49.971 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:49.972 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:49.972 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:49.972 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:49.972 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:49.973 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:49.973 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:49.973 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:49.973 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:49.973 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:49.974 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:49.974 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:49.974 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:49.974 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:49.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:49.975 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:49.975 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:49.975 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:49.976 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:49.976 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:54:49.976 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:49.976 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:49.977 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:49.977 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:54:49.977 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:49.977 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:49.977 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:49.978 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:49.978 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:49.978 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:49.978 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:49.978 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:49.979 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:49.980 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:49.980 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:49.980 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:49.980 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:49.980 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:49.981 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:49.981 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:49.981 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:49.981 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:49.981 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:49.982 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T11:54:54.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:54.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:54:54.985 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:54:55.030 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:54:55.030 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:54:55.096 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:54:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:54:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:54:55.097 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:54:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:54:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:54:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:55.098 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:54:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:54:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:55.099 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:54:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:54:55.100 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:54:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:55.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:54:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:55.102 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:54:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:55.103 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:54:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:55.104 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:54:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.561874_163.1", 2021-09-06T11:54:55.105 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.561874Z", 2021-09-06T11:54:55.106 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:54:55.107 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:54:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:54:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:54:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:54:55.108 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:54:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:54:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:54:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:54:55.109 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:54:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:54:55.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:54:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:54:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:54:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:54:55.111 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:54:55.112 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:54:55.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.561874_163.1 2021-09-06T11:55:00.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:00.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:00.115 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:00.175 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:00.251 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:00.265 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:00.265 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:00.265 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:00.265 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:00.266 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:00.266 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:00.266 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:00.266 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:00.267 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:00.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:00.267 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:00.267 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:00.267 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:00.268 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:00.268 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:00.268 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:00.268 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:00.269 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:00.269 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:00.269 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:00.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:00.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:00.270 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:00.270 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:00.270 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:00.271 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:00.271 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:00.271 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:00.271 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:00.272 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:00.272 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:00.272 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:00.272 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:00.272 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:00.273 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:00.273 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:00.273 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:00.273 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:55:00.274 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:00.274 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:00.274 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:00.274 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:55:00.275 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:00.275 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:00.275 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:00.275 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:00.276 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:00.276 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:00.276 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:00.276 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:00.276 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:00.277 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:00.277 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:00.277 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:00.277 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:00.278 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:00.278 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:00.278 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:00.278 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:00.278 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:00.279 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:00.279 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:00.279 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:00.280 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:00.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.738114_171.1 2021-09-06T11:55:05.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:05.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:05.284 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:05.347 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:05.348 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:05.520 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:05.521 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:05.522 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:05.523 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:05.524 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:05.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:05.526 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:05.527 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:05.528 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.797272_178.1", 2021-09-06T11:55:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:05.529 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:05.530 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.797272Z", 2021-09-06T11:55:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:05.531 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:05.532 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:05.533 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:05.534 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:05.535 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:05.536 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:05.536 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.797272_178.1 2021-09-06T11:55:10.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:10.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:10.539 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:10.588 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:10.589 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:10.960 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:10.961 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:10.961 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:10.961 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:10.961 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:10.962 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:10.962 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:10.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:10.962 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:10.963 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:10.963 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:10.963 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:10.963 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:10.963 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:10.964 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:10.964 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:10.964 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:10.964 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:10.964 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:10.965 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:10.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:10.965 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:10.965 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:10.965 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:10.966 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:10.966 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:10.966 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:10.966 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:10.967 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:10.967 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:10.967 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:10.967 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:10.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:10.968 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:10.968 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:10.968 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:10.968 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:10.969 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.615577_166.1", 2021-09-06T11:55:10.969 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:10.969 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:10.969 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:10.969 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.615577Z", 2021-09-06T11:55:10.970 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:10.970 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:10.970 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:10.970 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:10.970 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:10.971 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:10.971 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:10.971 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:10.971 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:10.971 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:10.972 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:10.972 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:10.972 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:10.972 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:10.972 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:10.973 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:10.973 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:10.973 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:10.973 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:10.974 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:10.974 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:10.974 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:10.974 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.615577_166.1 2021-09-06T11:55:15.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:15.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:15.977 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:16.020 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:16.020 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:16.093 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:16.093 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:16.094 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:16.094 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:16.094 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:16.094 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:16.095 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:16.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:16.095 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:16.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:16.096 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:16.096 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:16.096 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:16.096 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:16.097 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:16.097 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:16.097 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:16.097 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:16.097 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:16.098 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:16.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:16.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:16.098 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:16.099 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:16.099 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:16.099 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:16.099 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:16.099 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:16.100 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:16.100 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:16.100 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:16.100 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:16.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:16.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:16.101 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:16.101 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:16.101 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:16.102 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.676443_168.1", 2021-09-06T11:55:16.102 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:16.102 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:16.102 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:16.102 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.676443Z", 2021-09-06T11:55:16.103 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:16.103 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:16.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:16.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:16.104 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:16.104 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:16.104 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:16.104 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:16.104 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:16.105 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:16.105 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:16.105 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:16.105 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:16.105 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:16.106 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:16.106 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:16.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:16.106 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:16.106 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:16.107 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:16.107 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:16.107 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:16.108 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.676443_168.1 2021-09-06T11:55:21.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:21.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:21.111 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:21.161 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:21.162 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:21.234 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:21.234 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:21.235 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:21.235 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:21.235 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:21.235 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:21.236 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:21.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:21.236 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:21.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:21.237 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:21.237 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:21.237 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:21.237 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:21.237 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:21.238 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:21.238 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:21.238 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:21.238 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:21.239 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:21.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:21.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:21.239 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:21.240 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:21.240 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:21.240 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:21.240 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:21.240 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:21.241 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:21.241 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:21.241 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:21.241 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:21.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:21.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:21.242 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:21.242 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:21.242 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:21.243 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:55:21.243 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:21.243 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:21.243 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:21.243 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:55:21.244 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:21.244 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:21.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:21.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:21.245 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:21.245 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:21.245 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:21.245 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:21.245 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:21.246 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:21.246 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:21.246 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:21.246 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:21.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:21.247 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:21.247 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:21.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:21.247 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:21.247 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:21.248 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:21.248 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:21.248 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:21.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.507089_156.1 2021-09-06T11:55:26.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:26.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:26.256 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:26.323 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:26.323 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:26.420 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:26.420 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:26.420 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:26.421 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:26.421 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:26.421 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:26.421 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:26.421 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:26.422 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:26.422 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:26.422 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:26.422 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:26.422 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:26.423 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:26.423 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:26.423 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:26.423 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:26.424 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:26.424 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:26.424 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:26.424 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:26.424 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:26.425 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:26.425 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:26.425 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:26.425 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:26.425 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:26.426 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:26.426 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:26.426 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:26.426 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:26.427 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:26.427 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:26.427 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:26.427 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:26.428 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:26.428 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:26.428 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.736870_170.1", 2021-09-06T11:55:26.428 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:26.428 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:26.429 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:26.429 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.736870Z", 2021-09-06T11:55:26.429 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:26.429 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:26.429 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:26.430 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:26.430 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:26.430 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:26.430 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:26.431 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:26.431 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:26.431 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:26.431 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:26.431 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:26.432 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:26.432 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:26.432 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:26.432 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:26.433 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:26.433 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:26.433 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:26.433 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:26.433 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:26.434 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:26.435 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.736870_170.1 2021-09-06T11:55:31.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:31.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:31.437 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:31.502 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:31.502 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:32.156 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:32.157 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:32.157 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:32.157 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:32.157 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:32.158 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:32.158 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:32.158 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:32.158 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:32.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:32.159 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:32.159 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:32.159 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:32.160 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:32.160 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:32.160 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:32.160 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:32.161 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:32.161 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:32.161 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:32.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:32.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:32.162 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:32.162 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:32.162 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:32.162 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:32.163 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:32.163 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:32.163 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:32.163 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:32.164 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:32.164 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:32.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:32.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:32.164 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:32.165 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:32.165 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:32.165 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.539819_159.1", 2021-09-06T11:55:32.165 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:32.166 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:32.166 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:32.166 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.539819Z", 2021-09-06T11:55:32.166 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:32.167 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:32.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:32.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:32.167 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:32.168 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:32.168 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:32.168 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:32.168 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:32.169 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:32.169 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:32.169 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:32.169 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:32.169 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:32.170 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:32.170 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:32.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:32.170 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:32.171 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:32.171 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:32.171 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:32.171 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:32.172 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.539819_159.1 2021-09-06T11:55:37.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:37.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:37.174 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:37.274 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:37.274 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:37.694 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:37.694 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:37.695 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:37.695 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:37.695 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:37.695 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:37.696 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:37.696 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:37.696 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:37.696 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:37.697 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:37.697 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:37.697 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:37.697 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:37.698 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:37.698 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:37.698 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:37.698 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:37.699 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:37.699 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:37.699 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:37.699 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:37.700 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:37.700 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:37.700 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:37.700 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:37.700 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:37.701 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:37.701 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:37.701 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:37.701 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:37.701 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:37.702 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:37.702 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:37.702 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:37.702 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:37.703 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:37.703 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:55:37.703 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:37.703 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:37.704 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:37.704 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:55:37.704 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:37.704 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:37.704 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:37.705 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:37.705 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:37.705 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:37.705 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:37.706 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:37.706 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:37.706 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:37.706 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:37.707 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:37.707 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:37.707 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:37.707 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:37.708 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:37.708 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:37.708 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:37.708 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:37.708 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:37.709 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:37.709 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:37.710 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T11:55:42.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:42.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:42.713 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:42.768 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:42.768 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:43.128 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:43.128 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:43.128 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:43.128 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:43.129 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:43.129 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:43.129 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:43.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:43.130 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:43.130 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:43.130 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:43.130 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:43.131 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:43.131 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:43.131 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:43.131 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:43.131 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:43.132 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:43.132 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:43.132 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:43.132 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:43.132 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:43.133 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:43.133 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:43.133 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:43.133 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:43.133 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:43.134 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:43.134 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:43.134 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:43.134 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:43.135 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:43.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:43.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:43.135 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:43.136 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:43.136 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:43.136 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.615577_166.1", 2021-09-06T11:55:43.136 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:43.137 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:43.137 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:43.137 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.615577Z", 2021-09-06T11:55:43.138 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:43.138 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:43.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:43.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:43.139 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:43.139 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:43.139 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:43.139 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:43.140 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:43.140 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:43.140 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:43.140 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:43.141 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:43.141 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:43.141 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:43.141 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:43.142 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:43.142 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:43.142 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:43.142 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:43.143 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:43.143 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:43.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.615577_166.1 2021-09-06T11:55:48.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:48.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:48.147 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:48.195 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:48.195 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:48.269 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:55:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:55:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:48.284 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:48.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.503662_155.1 2021-09-06T11:55:53.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:53.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:53.288 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:53.330 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:53.330 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:53.402 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:53.402 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:53.403 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:53.405 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:53.406 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:53.407 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:53.408 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:53.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:53.410 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:55:53.411 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:53.411 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:53.411 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:53.411 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:55:53.412 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:53.412 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:53.412 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:53.412 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:53.413 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:53.413 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:53.413 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:53.413 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:53.413 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:53.414 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:53.414 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:53.414 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:53.414 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:53.415 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:53.415 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:53.415 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:53.415 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:53.416 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:53.416 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:53.416 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:53.416 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:53.417 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:53.418 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T11:55:58.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:58.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:55:58.421 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:55:58.490 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:55:58.490 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:55:58.954 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:55:58.955 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:55:58.955 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:55:58.956 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:55:58.956 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:55:58.956 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:55:58.956 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:58.957 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:58.957 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:55:58.957 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:58.957 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:55:58.958 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:58.958 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:58.958 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:55:58.958 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:58.959 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:55:58.959 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:58.959 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:55:58.959 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:58.959 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:58.960 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:58.960 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:58.960 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:55:58.960 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:58.961 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:58.961 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:55:58.961 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:58.961 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:58.961 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:58.962 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:55:58.962 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:58.962 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:58.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:58.963 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:58.963 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:55:58.963 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:58.964 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:58.964 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:55:58.964 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:58.964 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:58.964 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:58.965 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:55:58.965 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:58.965 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:58.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:55:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:55:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:55:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:55:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:55:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:55:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:55:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:55:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:55:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:55:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:55:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:55:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:55:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:55:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:55:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:55:58.970 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:55:58.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.738114_171.1 2021-09-06T11:56:03.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:03.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:03.973 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:04.063 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:04.063 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:04.245 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:04.250 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:04.250 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:04.250 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.543029_160.1", 2021-09-06T11:56:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.543029Z", 2021-09-06T11:56:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:04.260 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:04.260 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:04.260 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:04.261 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.543029_160.1 2021-09-06T11:56:09.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:09.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:09.264 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:09.315 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:09.316 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:09.392 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:09.392 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:09.392 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:09.393 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:09.393 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:09.393 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:09.394 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:09.394 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:09.394 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:09.394 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:09.395 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:09.395 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:09.395 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:09.395 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:09.395 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:09.396 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:09.396 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:09.396 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:09.396 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:09.397 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:09.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:09.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:09.397 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:09.397 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:09.398 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:09.398 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:09.398 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:09.398 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:09.399 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:09.399 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:09.399 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:09.399 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:09.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:09.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:09.400 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:09.400 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:09.400 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:09.400 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.561874_163.1", 2021-09-06T11:56:09.401 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:09.401 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:09.401 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:09.401 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.561874Z", 2021-09-06T11:56:09.402 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:09.402 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:09.402 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:09.402 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:09.403 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:09.403 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:09.403 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:09.403 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:09.403 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:09.404 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:09.404 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:09.404 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:09.404 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:09.405 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:09.405 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:09.405 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:09.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:09.405 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:09.406 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:09.406 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:09.406 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:09.406 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:09.407 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.561874_163.1 2021-09-06T11:56:14.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:14.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:14.410 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:14.452 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:14.453 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:14.522 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:14.523 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:14.523 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:14.523 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:14.524 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:14.524 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:14.524 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:14.524 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:14.525 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:14.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:14.525 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:14.525 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:14.525 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:14.526 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:14.526 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:14.526 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:14.526 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:14.527 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:14.527 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:14.527 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:14.527 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:14.528 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:14.528 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:14.528 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:14.528 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:14.529 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:14.529 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:14.529 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:14.529 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:14.529 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:14.530 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:14.530 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:14.530 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:14.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:14.531 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:14.531 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:14.531 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:14.531 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.543029_160.1", 2021-09-06T11:56:14.531 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:14.532 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:14.532 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:14.532 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.543029Z", 2021-09-06T11:56:14.532 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:14.533 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:14.533 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:14.533 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:14.533 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:14.534 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:14.534 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:14.534 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:14.534 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:14.534 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:14.535 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:14.535 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:14.535 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:14.535 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:14.536 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:14.536 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:14.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:14.536 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:14.536 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:14.537 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:14.537 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:14.537 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:14.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.543029_160.1 2021-09-06T11:56:19.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:19.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:19.541 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:19.624 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:19.624 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:20.051 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:20.052 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:20.052 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:20.052 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:20.052 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:20.060 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:20.060 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:20.060 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.827863_182.1", 2021-09-06T11:56:20.060 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:20.061 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:20.061 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:20.061 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.827863Z", 2021-09-06T11:56:20.061 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:20.061 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:20.066 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:20.066 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:20.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.827863_182.1 2021-09-06T11:56:25.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:25.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:25.069 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:25.122 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:25.122 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:25.245 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:25.245 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:25.245 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:25.245 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:25.246 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:25.246 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:25.246 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:25.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:25.247 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:25.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:25.247 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:25.247 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:25.248 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:25.248 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:25.248 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:25.248 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:25.248 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:25.249 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:25.249 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:25.249 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:25.249 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:25.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:25.250 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:25.250 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:25.250 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:25.250 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:25.250 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:25.251 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:25.251 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:25.251 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:25.251 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:25.252 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:25.252 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:25.252 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:25.252 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:25.252 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:25.253 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:25.253 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:56:25.253 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:25.253 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:25.254 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:25.254 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:56:25.254 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:25.254 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:25.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:25.255 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:25.255 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:25.255 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:25.255 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:25.256 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:25.256 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:25.256 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:25.256 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:25.257 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:25.257 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:25.257 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:25.257 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:25.257 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:25.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:25.258 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:25.258 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:25.258 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:25.259 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:25.259 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:25.260 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.503662_155.1 2021-09-06T11:56:30.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:30.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:30.262 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:30.335 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:30.335 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:30.963 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:30.964 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:30.964 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:30.964 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:30.965 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:30.965 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:30.965 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:30.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:30.966 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:30.966 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:30.966 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:30.966 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:30.967 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:30.967 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:30.967 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:30.968 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:30.968 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:30.968 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:30.968 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:30.969 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:30.969 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:30.969 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:30.969 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:30.969 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:30.970 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:30.970 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:30.970 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:30.970 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:30.971 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:30.971 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:30.971 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:30.971 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:30.972 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:30.972 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:30.972 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:30.972 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:30.972 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:30.973 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T11:56:30.973 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:30.973 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:30.973 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:30.974 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T11:56:30.974 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:30.974 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:30.974 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:30.975 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:30.975 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:30.975 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:30.975 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:30.975 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:30.976 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:30.976 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:30.976 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:30.977 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:30.977 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:30.977 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:30.978 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:30.978 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:30.978 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:30.978 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:30.979 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:30.979 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:30.979 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:30.979 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:30.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.489232_154.1 2021-09-06T11:56:35.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:35.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:35.982 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:36.031 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:36.032 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:36.104 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:36.104 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:36.105 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:36.105 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:36.105 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:36.105 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:36.106 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:36.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:36.106 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:36.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:36.107 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:36.107 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:36.107 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:36.107 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:36.108 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:36.108 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:36.108 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:36.108 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:36.108 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:36.109 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:36.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:36.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:36.109 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:36.110 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:36.110 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:36.110 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:36.110 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:36.111 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:36.111 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:36.111 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:36.111 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:36.112 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:36.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:36.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:36.112 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:36.113 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:36.113 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:36.113 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:56:36.113 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:36.114 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:36.114 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:36.114 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:56:36.114 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:36.115 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:36.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:36.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:36.116 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:36.116 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:36.116 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:36.116 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:36.116 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:36.117 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:36.117 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:36.117 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:36.117 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:36.118 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:36.118 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:36.118 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:36.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:36.119 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:36.119 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:36.119 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:36.119 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:36.120 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:36.121 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.507089_156.1 2021-09-06T11:56:41.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:41.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:41.124 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:41.167 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:41.167 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:41.232 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:41.233 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:41.233 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:41.233 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:41.233 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:41.234 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:41.234 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:41.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:41.234 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:41.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:41.235 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:41.235 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:41.235 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:41.235 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:41.236 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:41.236 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:41.236 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:41.236 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:41.236 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:41.237 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:41.237 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:41.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:41.237 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:41.237 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:41.238 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:41.238 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:41.238 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:41.238 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:41.239 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:41.239 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:41.239 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:41.239 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:41.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:41.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:41.240 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:41.240 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:41.240 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:41.240 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:56:41.241 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:41.241 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:41.241 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:41.241 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:56:41.242 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:41.242 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:41.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:41.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:41.242 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:41.243 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:41.243 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:41.243 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:41.243 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:41.244 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:41.244 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:41.244 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:41.244 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:41.245 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:41.245 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:41.245 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:41.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:41.245 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:41.246 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:41.246 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:41.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:41.246 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:41.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.508454_157.1 2021-09-06T11:56:46.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:46.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:46.250 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:46.343 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:46.343 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:46.800 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:46.801 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:46.801 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:46.801 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:46.801 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:46.802 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:46.802 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:46.802 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:46.802 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:46.803 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:46.803 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:46.803 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:46.803 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:46.804 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:46.804 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:46.804 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:46.804 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:46.805 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:46.805 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:46.805 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:46.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:46.805 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:46.806 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:46.806 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:46.806 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:46.807 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:46.807 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:46.807 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:46.807 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:46.807 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:46.808 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:46.808 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:46.808 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:46.808 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:46.809 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:46.809 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:46.809 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:46.809 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:56:46.809 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:46.810 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:46.810 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:46.810 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:56:46.810 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:46.811 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:46.811 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:46.811 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:46.811 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:46.812 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:46.812 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:46.812 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:46.812 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:46.812 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:46.813 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:46.813 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:46.813 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:46.813 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:46.813 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:46.814 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:46.814 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:46.814 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:46.814 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:46.815 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:46.815 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:46.815 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:46.816 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T11:56:51.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:51.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:51.818 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:51.897 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:51.897 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:52.018 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:52.018 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:52.018 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:52.019 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:52.019 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:52.019 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:52.019 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:52.020 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:52.020 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:52.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:52.020 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:52.021 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:52.021 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:52.021 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:52.021 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:52.021 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:52.022 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:52.022 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:52.022 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:52.022 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:52.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:52.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:52.023 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:52.023 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:52.023 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:52.024 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:52.024 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:52.024 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:52.024 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:52.024 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:52.025 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:52.025 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:52.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:52.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:52.026 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:52.026 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:52.026 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:52.026 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.758635_174.1", 2021-09-06T11:56:52.026 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:52.027 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:52.027 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:52.027 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.758635Z", 2021-09-06T11:56:52.027 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:52.028 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:52.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:52.028 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:52.028 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:52.029 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:52.029 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:52.029 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:52.029 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:52.030 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:52.030 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:52.030 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:52.030 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:52.030 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:52.031 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:52.031 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:52.031 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:52.031 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:52.031 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:52.032 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:52.032 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:52.032 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:52.033 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.758635_174.1 2021-09-06T11:56:57.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:57.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:56:57.036 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:56:57.096 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:56:57.096 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:56:57.687 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:56:57.688 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:56:57.688 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:56:57.688 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:56:57.689 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:56:57.689 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:56:57.689 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:57.689 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:57.689 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:56:57.690 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:57.690 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:56:57.690 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:57.690 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:57.691 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:56:57.691 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:57.691 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:56:57.691 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:57.691 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:56:57.692 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:57.692 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:57.692 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:57.692 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:57.692 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:56:57.693 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:57.693 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:57.693 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:56:57.693 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:57.693 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:57.694 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:57.694 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:56:57.694 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:57.694 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:57.694 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:57.695 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:57.695 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:56:57.695 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:57.695 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:57.696 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:56:57.696 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:57.696 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:57.696 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:57.696 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:56:57.697 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:57.697 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:57.697 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:57.697 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:56:57.697 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:56:57.698 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:56:57.698 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:56:57.698 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:56:57.698 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:56:57.699 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:56:57.699 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:56:57.699 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:56:57.699 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:56:57.699 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:57.700 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:57.700 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:56:57.700 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:56:57.700 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:56:57.700 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:56:57.701 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:56:57.701 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:56:57.701 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:56:57.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.508454_157.1 2021-09-06T11:57:02.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:02.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:02.705 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:02.753 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:02.753 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:02.831 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:02.831 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:02.832 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:02.833 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:02.834 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:02.835 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:02.836 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:02.837 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:02.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:02.839 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:57:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:02.840 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:57:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:02.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:02.842 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:02.843 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:02.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:02.845 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:02.845 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:02.847 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.503662_155.1 2021-09-06T11:57:07.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:07.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:07.849 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:07.894 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:07.895 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:07.962 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:07.962 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:07.962 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:07.963 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:07.963 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:07.963 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:07.964 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:07.964 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:07.964 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:07.964 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:07.964 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:07.965 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:07.965 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:07.965 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:07.965 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:07.966 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:07.966 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:07.966 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:07.966 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:07.966 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:07.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:07.967 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:07.967 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:07.967 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:07.968 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:07.968 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:07.968 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:07.968 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:07.969 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:07.969 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:07.969 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:07.969 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:07.970 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:07.970 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:07.970 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:07.970 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:07.970 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:07.971 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.615577_166.1", 2021-09-06T11:57:07.971 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:07.971 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:07.971 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:07.972 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.615577Z", 2021-09-06T11:57:07.972 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:07.972 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:07.972 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:07.972 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:07.973 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:07.973 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:07.973 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:07.973 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:07.974 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:07.974 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:07.974 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:07.974 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:07.975 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:07.975 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:07.975 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:07.975 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:07.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:07.976 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:07.976 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:07.976 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:07.976 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:07.977 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:07.978 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.615577_166.1 2021-09-06T11:57:12.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:12.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:12.980 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:13.054 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:13.054 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:13.161 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:13.161 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:13.162 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:13.162 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:13.162 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:13.162 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:13.163 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:13.163 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:13.163 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:13.163 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:13.164 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:13.164 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:13.164 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:13.164 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:13.164 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:13.165 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:13.165 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:13.165 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:13.165 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:13.166 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:13.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:13.166 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:13.166 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:13.166 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:13.167 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:13.167 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:13.167 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:13.167 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:13.168 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:13.168 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:13.168 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:13.168 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:13.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:13.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:13.169 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:13.169 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:13.169 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:13.169 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.736870_170.1", 2021-09-06T11:57:13.170 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:13.170 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:13.170 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:13.170 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.736870Z", 2021-09-06T11:57:13.171 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:13.171 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:13.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:13.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:13.172 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:13.172 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:13.172 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:13.172 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:13.172 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:13.173 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:13.173 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:13.173 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:13.173 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:13.174 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:13.174 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:13.174 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:13.174 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:13.175 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:13.175 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:13.175 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:13.175 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:13.176 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:13.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.736870_170.1 2021-09-06T11:57:18.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2021-09-06T11:57:18.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:18.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:18.181 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:18.282 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:18.283 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:18.490 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=06970c73-91a9-4240-80e4-aa07d550699b 2021-09-06T11:57:18.505 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2021-09-06T11:57:18.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:57:18.506 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:18.506 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:25:14.716931Z" 2021-09-06T11:57:18.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:57:18.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:57:18.507 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "", 2021-09-06T11:57:18.507 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "0.000000" 2021-09-06T11:57:18.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:57:18.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:57:18.508 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630929121.899863_275.1", 2021-09-06T11:57:18.508 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:52:01.899863Z" 2021-09-06T11:57:18.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T11:57:18.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T11:57:18.509 INFO:teuthology.orchestra.run.smithi094.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:18.509 INFO:teuthology.orchestra.run.smithi094.stdout: "last_update": "2021-09-06T11:27:39.337096Z" 2021-09-06T11:57:18.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T11:57:18.509 INFO:teuthology.orchestra.run.smithi094.stdout:] 2021-09-06T11:57:18.510 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T11:57:18.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:18.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:18.512 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:18.598 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:18.598 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:19.026 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:19.026 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:19.026 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:19.027 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:19.027 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:19.027 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:19.027 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:19.028 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:19.028 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:19.028 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:19.028 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:19.028 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:19.029 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:19.029 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:19.029 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:19.029 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:19.029 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:19.030 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:19.030 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:19.030 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:19.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:19.031 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:19.031 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:19.031 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:19.031 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:19.031 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:19.032 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:19.032 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:19.032 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:19.032 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:19.033 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:19.033 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:19.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:19.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:19.033 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:19.034 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:19.034 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:19.034 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.615577_166.1", 2021-09-06T11:57:19.034 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:19.035 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:19.035 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:19.035 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.615577Z", 2021-09-06T11:57:19.035 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:19.035 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:19.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:19.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:19.036 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:19.036 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:19.036 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:19.037 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:19.037 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:19.037 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:19.038 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:19.038 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:19.038 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:19.038 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:19.038 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:19.039 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:19.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:19.039 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:19.039 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:19.039 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:19.040 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:19.040 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:19.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.615577_166.1 2021-09-06T11:57:24.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:24.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:24.044 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:24.133 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:24.133 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:24.283 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:24.283 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:24.283 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:24.284 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:24.284 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:24.284 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:24.284 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:24.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:24.285 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:24.285 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:24.285 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:24.286 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:24.286 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:24.286 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:24.286 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:24.287 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:24.287 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:24.287 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:24.287 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:24.287 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:24.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:24.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:24.288 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:24.288 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:24.289 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:24.289 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:24.289 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:24.289 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:24.290 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:24.290 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:24.290 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:24.290 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:24.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:24.291 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:24.291 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:24.291 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:24.291 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:24.292 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:57:24.292 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:24.292 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:24.292 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:24.293 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:57:24.293 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:24.293 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:24.293 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:24.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:24.294 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:24.294 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:24.294 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:24.294 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:24.295 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:24.295 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:24.295 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:24.295 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:24.296 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:24.296 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:24.296 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:24.296 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:24.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:24.297 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:24.297 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:24.297 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:24.297 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:24.297 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:24.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.507089_156.1 2021-09-06T11:57:29.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:29.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:29.301 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:29.350 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:29.351 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:29.432 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:29.432 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:29.432 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:29.433 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:29.433 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:29.433 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:29.433 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.561874_163.1", 2021-09-06T11:57:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.561874Z", 2021-09-06T11:57:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:29.445 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:29.445 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:29.445 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:29.445 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:29.446 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.561874_163.1 2021-09-06T11:57:34.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:34.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:34.449 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:34.491 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:34.492 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:34.557 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:34.558 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:34.558 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:34.558 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:34.558 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:34.559 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:34.559 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:34.559 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:34.559 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:34.559 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:34.560 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:34.560 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:34.560 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:34.560 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:34.561 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:34.561 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:34.561 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:34.561 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:34.561 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:34.562 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:34.562 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:34.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:34.562 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:34.562 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:34.563 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:34.563 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:34.563 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:34.563 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:34.563 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:34.564 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:34.564 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:34.564 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:34.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:34.565 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:34.565 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:34.565 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:34.565 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:34.565 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.561874_163.1", 2021-09-06T11:57:34.566 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:34.566 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:34.566 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:34.566 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.561874Z", 2021-09-06T11:57:34.566 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:34.567 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:34.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:34.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:34.567 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:34.567 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:34.568 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:34.568 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:34.568 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:34.568 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:34.568 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:34.569 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:34.569 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:34.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:34.569 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:34.569 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:34.570 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:34.570 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:34.570 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:34.570 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:34.570 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:34.571 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:34.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.561874_163.1 2021-09-06T11:57:39.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:39.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:39.574 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:39.665 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:39.666 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:40.055 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:40.055 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:40.056 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:40.056 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:40.056 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:40.056 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:40.057 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:40.057 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:40.057 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:40.057 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:40.058 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:40.058 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:40.058 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:40.058 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:40.058 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:40.059 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:40.059 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:40.059 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:40.059 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:40.060 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:40.060 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:40.060 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:40.060 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:40.060 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:40.061 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:40.061 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:40.061 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:40.061 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:40.062 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:40.062 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:40.062 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:40.062 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:40.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:40.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:40.063 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:40.063 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:40.063 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:40.064 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:57:40.064 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:40.064 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:40.064 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:40.064 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:57:40.065 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:40.065 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:40.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:40.065 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:40.065 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:40.066 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:40.066 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:40.066 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:40.066 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:40.067 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:40.067 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:40.067 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:40.067 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:40.067 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:40.068 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:40.068 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:40.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:40.068 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:40.069 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:40.069 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:40.069 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:40.069 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:40.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T11:57:45.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:45.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:45.073 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:45.123 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:45.123 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:45.272 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:45.272 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:45.273 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:45.273 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:45.273 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:45.273 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:45.274 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:45.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:45.274 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:45.275 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:45.275 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:45.275 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:45.275 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:45.276 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:45.276 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:45.276 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:45.276 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:45.277 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:45.277 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:45.277 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:45.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:45.277 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:45.278 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:45.278 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:45.278 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:45.278 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:45.279 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:45.279 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:45.279 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:45.279 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:45.280 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:45.280 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:45.280 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:45.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:45.281 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:45.281 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:45.281 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:45.281 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:57:45.281 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:45.282 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:45.282 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:45.282 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:57:45.282 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:45.283 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:45.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:45.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:45.283 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:45.284 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:45.284 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:45.284 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:45.284 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:45.285 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:45.285 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:45.285 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:45.285 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:45.286 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:45.286 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:45.286 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:45.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:45.287 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:45.287 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:45.287 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:45.287 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:45.288 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:45.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.544174_161.1 2021-09-06T11:57:50.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:50.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:50.292 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:50.349 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:50.350 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:50.916 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:50.916 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:50.917 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:50.917 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:50.917 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:50.917 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:50.918 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:50.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:50.918 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:50.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:50.919 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:50.919 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:50.919 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:50.919 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:50.919 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:50.920 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:50.920 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:50.920 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:50.920 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:50.921 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:50.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:50.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:50.921 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:50.922 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:50.922 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:50.922 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:50.922 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:50.922 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:50.923 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:50.923 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:50.923 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:50.923 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:50.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:50.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:50.924 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:50.924 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:50.924 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:50.925 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.547334_162.1", 2021-09-06T11:57:50.925 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:50.925 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:50.925 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:50.926 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.547334Z", 2021-09-06T11:57:50.926 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:50.926 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:50.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:50.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:50.927 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:50.927 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:50.927 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:50.927 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:50.928 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:50.928 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:50.928 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:50.928 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:50.929 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:50.929 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:50.929 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:50.929 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:50.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:50.930 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:50.930 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:50.930 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:50.930 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:50.931 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:50.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.547334_162.1 2021-09-06T11:57:55.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:55.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:57:55.935 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:57:55.985 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:57:55.986 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:57:56.059 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:57:56.059 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:57:56.060 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:57:56.060 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:57:56.060 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:57:56.060 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:57:56.061 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:56.061 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:56.061 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:57:56.061 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:56.062 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:57:56.062 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:56.062 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:56.062 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:57:56.063 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:56.063 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:57:56.063 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:56.063 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:57:56.063 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:56.064 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:56.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:56.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:56.064 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:57:56.065 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:56.065 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:56.065 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:57:56.065 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:56.065 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:56.066 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:56.066 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:57:56.066 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:56.066 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:56.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:56.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:56.067 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:57:56.067 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:56.068 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:56.068 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:57:56.068 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:56.068 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:56.069 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:56.069 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:57:56.069 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:56.069 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:56.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:56.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:57:56.070 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:57:56.070 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:57:56.070 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:57:56.071 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:57:56.071 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:57:56.071 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:57:56.071 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:57:56.072 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:57:56.072 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:57:56.072 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:56.072 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:56.073 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:57:56.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:57:56.073 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:57:56.073 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:57:56.074 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:57:56.074 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:57:56.074 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:57:56.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.738114_171.1 2021-09-06T11:58:01.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:01.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:01.078 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:01.121 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:01.121 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:01.196 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:01.196 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:01.197 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:01.197 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:01.197 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:01.197 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:01.198 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:01.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:01.198 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:01.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:01.199 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:01.199 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:01.199 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:01.199 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:01.199 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:01.200 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:01.200 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:01.200 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:01.200 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:01.201 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:01.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:01.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:01.201 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:01.202 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:01.202 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:01.202 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:01.202 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:01.202 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:01.203 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:01.203 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:01.203 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:01.203 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:01.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:01.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:01.204 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:01.204 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:01.205 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:01.205 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.615577_166.1", 2021-09-06T11:58:01.205 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:01.205 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:01.206 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:01.206 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.615577Z", 2021-09-06T11:58:01.206 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:01.206 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:01.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:01.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:01.207 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:01.207 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:01.207 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:01.208 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:01.208 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:01.208 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:01.208 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:01.209 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:01.209 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:01.209 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:01.209 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:01.210 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:01.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:01.210 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:01.210 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:01.210 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:01.211 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:01.211 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:01.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.615577_166.1 2021-09-06T11:58:06.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:06.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:06.216 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:06.294 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:06.295 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:06.685 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:06.686 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:06.686 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:06.686 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:06.686 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:06.687 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:06.687 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:06.687 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:06.687 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:06.688 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:06.688 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:06.688 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:06.688 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:06.689 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:06.689 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:06.689 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:06.689 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:06.689 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:06.690 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:06.690 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:06.690 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:06.690 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:06.691 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:06.691 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:06.691 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:06.691 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:06.692 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:06.692 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:06.692 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:06.692 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:06.693 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:06.693 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:06.693 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:06.693 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:06.693 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:06.694 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:06.694 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:06.694 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:58:06.695 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:06.695 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:06.695 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:06.696 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:58:06.696 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:06.696 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:06.696 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:06.696 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:06.697 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:06.697 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:06.697 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:06.697 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:06.698 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:06.698 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:06.698 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:06.698 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:06.699 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:06.699 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:06.699 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:06.699 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:06.699 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:06.700 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:06.700 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:06.700 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:06.700 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:06.701 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:06.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.508454_157.1 2021-09-06T11:58:11.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:11.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:11.704 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:11.760 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:11.760 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:11.828 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:11.828 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:11.828 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:11.829 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:11.829 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:11.829 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:11.829 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:11.830 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:11.830 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:11.830 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:11.830 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:11.831 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:11.831 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:11.831 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:11.831 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:11.832 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:11.832 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:11.832 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:11.832 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:11.832 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:11.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:11.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:11.833 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:11.833 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:11.833 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:11.834 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:11.834 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:11.834 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:11.834 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:11.835 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:11.835 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:11.835 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:11.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:11.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:11.836 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:11.836 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:11.836 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:11.836 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:58:11.837 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:11.837 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:11.837 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:11.837 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:58:11.838 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:11.838 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:11.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:11.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:11.839 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:11.839 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:11.839 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:11.839 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:11.840 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:11.840 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:11.840 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:11.840 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:11.840 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:11.841 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:11.841 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:11.841 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:11.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:11.842 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:11.842 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:11.842 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:11.842 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:11.842 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:11.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.508454_157.1 2021-09-06T11:58:16.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:16.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:16.846 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:16.890 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:16.890 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:16.991 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:16.991 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:16.992 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:16.992 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:16.992 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:16.992 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:16.993 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:16.993 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:16.993 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:16.993 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:16.994 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:16.994 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:16.994 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:16.994 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:16.995 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:16.995 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:16.995 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:16.995 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:16.995 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:16.996 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:16.996 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:16.996 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:16.996 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:16.997 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:16.997 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:16.997 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:16.997 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:16.998 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:16.998 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:16.998 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:16.998 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:16.998 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:16.999 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:16.999 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:16.999 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:16.999 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:17.000 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:17.000 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.508454_157.1", 2021-09-06T11:58:17.000 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:17.000 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:17.001 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:17.001 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.508454Z", 2021-09-06T11:58:17.001 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:17.001 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:17.002 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:17.002 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:17.002 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:17.002 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:17.003 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:17.003 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:17.003 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:17.003 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:17.003 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:17.004 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:17.004 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:17.004 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:17.004 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:17.005 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:17.005 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:17.005 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:17.006 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:17.006 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:17.006 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:17.006 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:17.007 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.508454_157.1 2021-09-06T11:58:22.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:22.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:22.010 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:22.060 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:22.133 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:22.133 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:22.134 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:22.134 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:22.134 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:22.134 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:22.135 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:22.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:22.135 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:22.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:22.136 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:22.136 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:22.136 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:22.136 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:22.137 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:22.137 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:22.137 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:22.137 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:22.138 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:22.138 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:22.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:22.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:22.139 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:22.139 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:22.139 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:22.139 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:22.139 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:22.140 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:22.140 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:22.140 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:22.140 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:22.141 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:22.141 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:22.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:22.141 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:22.142 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:22.142 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:22.142 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.676443_168.1", 2021-09-06T11:58:22.142 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:22.143 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:22.143 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:22.143 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.676443Z", 2021-09-06T11:58:22.143 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:22.144 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:22.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:22.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:22.144 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:22.145 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:22.146 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:22.146 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:22.146 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:22.146 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:22.147 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:22.147 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:22.147 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:22.147 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:22.148 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:22.148 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:22.148 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:22.148 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:22.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.676443_168.1 2021-09-06T11:58:27.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:27.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:27.152 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:27.222 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:27.222 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:27.446 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:27.447 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:27.447 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:27.447 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:27.447 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:27.448 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:27.448 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:27.448 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:27.448 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:27.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:27.449 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:27.449 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:27.449 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:27.450 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:27.450 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:27.450 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:27.450 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:27.450 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:27.451 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:27.451 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:27.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:27.451 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:27.452 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:27.452 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:27.452 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:27.452 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:27.453 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:27.453 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:27.453 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:27.453 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:27.453 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:27.454 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:27.454 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:27.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:27.454 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:27.455 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:27.455 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:27.455 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:58:27.455 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:27.456 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:27.456 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:27.456 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:58:27.456 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:27.456 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:27.457 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:27.457 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:27.457 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:27.457 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:27.458 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:27.458 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:27.458 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:27.458 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:27.458 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:27.459 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:27.459 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:27.459 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:27.460 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:27.460 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:27.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:27.460 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:27.460 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:27.461 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:27.461 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:27.461 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:27.462 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.507089_156.1 2021-09-06T11:58:32.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:32.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:32.466 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:32.521 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:32.521 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:32.616 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:32.617 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:32.617 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:32.617 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:32.618 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:32.618 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:32.618 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:32.618 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:32.619 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:32.619 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:32.619 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:32.619 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:32.620 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:32.620 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:32.620 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:32.620 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:32.621 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:32.621 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:32.621 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:32.621 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:32.621 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:32.622 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:32.622 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:32.622 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:32.622 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:32.623 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:32.623 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:32.623 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:32.623 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:32.623 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:32.624 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:32.624 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:32.624 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:32.624 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:32.625 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:32.625 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:32.625 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:32.625 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:58:32.625 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:32.626 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:32.626 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:32.626 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:58:32.626 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:32.627 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:32.627 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:32.627 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:32.627 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:32.628 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:32.628 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:32.628 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:32.628 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:32.628 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:32.629 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:32.629 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:32.629 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:32.629 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:32.630 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:32.630 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:32.630 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:32.630 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:32.631 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:32.631 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:32.631 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:32.631 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:32.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.507089_156.1 2021-09-06T11:58:37.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:37.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:37.635 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:37.726 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:37.726 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:38.162 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:38.162 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:38.163 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:38.163 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:38.163 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:38.163 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:38.164 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:38.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:38.164 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:38.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:38.165 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:38.165 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:38.165 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:38.165 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:38.165 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:38.166 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:38.166 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:38.166 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:38.166 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:38.167 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:38.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:38.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:38.167 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:38.167 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:38.168 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:38.168 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:38.168 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:38.168 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:38.169 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:38.169 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:38.169 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:38.169 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:38.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:38.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:38.170 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:38.170 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:38.170 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:38.171 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.759827_175.1", 2021-09-06T11:58:38.171 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:38.171 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:38.171 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:38.172 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.759827Z", 2021-09-06T11:58:38.172 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:38.172 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:38.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:38.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:38.173 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:38.173 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:38.173 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:38.174 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:38.174 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:38.174 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:38.174 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:38.175 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:38.175 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:38.175 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:38.175 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:38.176 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:38.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:38.176 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:38.176 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:38.176 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:38.177 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:38.177 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:38.178 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.759827_175.1 2021-09-06T11:58:43.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:43.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:43.181 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:43.232 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:43.232 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:43.322 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:43.322 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:43.323 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:43.323 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:43.323 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:43.323 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:43.324 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:43.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:43.324 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:43.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:43.325 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:43.325 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:43.325 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:43.325 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:43.325 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:43.326 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:43.326 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:43.326 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:43.326 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:43.327 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:43.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:43.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:43.327 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:43.328 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:43.328 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:43.328 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:43.328 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:43.329 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:43.329 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:43.329 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:43.329 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:43.329 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:43.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:43.330 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:43.330 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:43.330 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:43.331 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:43.331 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T11:58:43.331 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:43.331 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:43.332 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:43.332 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T11:58:43.332 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:43.332 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:43.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:43.333 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:43.333 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:43.333 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:43.333 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:43.334 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:43.334 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:43.334 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:43.334 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:43.335 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:43.335 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:43.335 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:43.335 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:43.335 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:43.336 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:43.336 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:43.336 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:43.336 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:43.337 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:43.337 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:43.338 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.503662_155.1 2021-09-06T11:58:48.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:48.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:48.342 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:48.385 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:48.386 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:48.461 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:48.462 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:48.462 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:48.462 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:48.463 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:48.463 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:48.463 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:48.463 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:48.464 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:48.464 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:48.464 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:48.464 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:48.465 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:48.465 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:48.465 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:48.465 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:48.465 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:48.466 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:48.466 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:48.466 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:48.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:48.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:48.467 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:48.467 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:48.467 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:48.468 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:48.468 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:48.468 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:48.468 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:48.468 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:48.469 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:48.469 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:48.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:48.470 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:48.470 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:48.470 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:48.471 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:48.471 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T11:58:48.471 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:48.471 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:48.472 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:48.472 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T11:58:48.472 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:48.472 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:48.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:48.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:48.473 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:48.473 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:48.473 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:48.474 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:48.474 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:48.474 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:48.474 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:48.475 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:48.475 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:48.475 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:48.475 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:48.476 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:48.476 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:48.476 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:48.476 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:48.476 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:48.477 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:48.477 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:48.478 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.507089_156.1 2021-09-06T11:58:53.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:53.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:53.481 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:53.569 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:53.570 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:53.838 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:53.839 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:53.839 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:53.839 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:53.839 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:53.839 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:53.840 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:53.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:53.840 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:53.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:53.841 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:53.841 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:53.841 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:53.841 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:53.841 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:53.842 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:53.842 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:53.842 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:53.842 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:53.842 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:53.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:53.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:53.843 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:53.843 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:53.844 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:53.844 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:53.844 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:53.844 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:53.844 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:53.845 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:53.845 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:53.845 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:53.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:53.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:53.846 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:53.846 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:53.846 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:53.846 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.676443_168.1", 2021-09-06T11:58:53.847 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:53.847 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:53.847 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:53.847 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.676443Z", 2021-09-06T11:58:53.847 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:53.848 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:53.848 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:53.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:53.848 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:53.848 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:53.849 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:53.849 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:53.849 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:53.849 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:53.849 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:53.850 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:53.850 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:53.850 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:53.850 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:53.851 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:53.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:53.851 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:53.851 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:53.851 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:53.852 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:53.852 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:53.852 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.676443_168.1 2021-09-06T11:58:58.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:58.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:58:58.855 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:58:58.897 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:58:58.898 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:58:58.965 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:58:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:58:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:58:58.966 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:58:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:58:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:58:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:58.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:58:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:58:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:58.968 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:58:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:58:58.969 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:58.970 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:58:58.970 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:58.970 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:58.970 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:58.970 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:58.971 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:58:58.971 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:58.971 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:58.971 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:58:58.972 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:58.972 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:58.972 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:58.972 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:58:58.973 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:58.973 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:58.973 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:58.973 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:58.973 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:58:58.974 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:58.974 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:58.974 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:58:58.974 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:58.975 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:58.975 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:58.975 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:58:58.975 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:58.975 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:58.976 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:58.976 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:58:58.976 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:58:58.976 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:58:58.977 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:58:58.977 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:58:58.977 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:58:58.977 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:58:58.977 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:58:58.978 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:58:58.978 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:58:58.978 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:58.978 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:58.978 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:58:58.979 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:58:58.979 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:58:58.979 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:58:58.979 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:58:58.980 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:58:58.980 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:58:58.981 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T11:59:03.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:03.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:03.984 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:04.055 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:04.055 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:04.703 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:04.703 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:04.703 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:04.704 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:04.704 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:04.704 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:04.704 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:04.705 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:04.705 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:04.705 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:04.705 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:04.706 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:04.706 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:04.706 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:04.706 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:04.706 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:04.707 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:04.707 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:04.707 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:04.707 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:04.708 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:04.708 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:04.708 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:04.709 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:04.709 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:04.709 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:04.709 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:04.709 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:04.710 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:04.710 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:04.710 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:04.710 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:04.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:04.711 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:04.711 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:04.711 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:04.711 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:04.712 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T11:59:04.712 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:04.712 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:04.712 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:04.712 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T11:59:04.713 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:04.713 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:04.713 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:04.714 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:04.714 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:04.714 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:04.714 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:04.715 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:04.715 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:04.715 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:04.715 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:04.715 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:04.716 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:04.716 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:04.716 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:04.716 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:04.716 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:04.717 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:04.717 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:04.717 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:04.717 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:04.718 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:04.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.544174_161.1 2021-09-06T11:59:09.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:09.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:09.722 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:09.777 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:09.777 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:10.268 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:10.268 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:10.269 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:10.269 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:10.269 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:10.269 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:10.269 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:10.270 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:10.270 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:10.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:10.270 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:10.271 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:10.271 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:10.271 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:10.271 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:10.272 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:10.272 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:10.272 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:10.272 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:10.273 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:10.273 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:10.273 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:10.273 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:10.273 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:10.274 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:10.274 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:10.274 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:10.275 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:10.275 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:10.275 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:10.275 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:10.275 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:10.276 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:10.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:10.276 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:10.276 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:10.277 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:10.277 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.547334_162.1", 2021-09-06T11:59:10.277 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:10.277 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:10.278 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:10.278 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.547334Z", 2021-09-06T11:59:10.278 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:10.278 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:10.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:10.279 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:10.279 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:10.280 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:10.280 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:10.280 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:10.280 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:10.281 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:10.281 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:10.281 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:10.281 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:10.282 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:10.282 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:10.282 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:10.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:10.283 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:10.283 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:10.283 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:10.283 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:10.284 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:10.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.547334_162.1 2021-09-06T11:59:15.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:15.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:15.287 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:15.334 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:15.334 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:15.417 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:15.418 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:15.418 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:15.418 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:15.419 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:15.419 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:15.419 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:15.419 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:15.419 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:15.420 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:15.420 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:15.420 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:15.420 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:15.421 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:15.421 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:15.421 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:15.421 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:15.422 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:15.422 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:15.422 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:15.422 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:15.423 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:15.423 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:15.423 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:15.423 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:15.423 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:15.424 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:15.424 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:15.424 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:15.424 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:15.425 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:15.425 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:15.425 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:15.425 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:15.425 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:15.426 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:15.426 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:15.426 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.733637_169.1", 2021-09-06T11:59:15.426 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:15.427 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:15.427 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:15.427 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.733637Z", 2021-09-06T11:59:15.427 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:15.428 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:15.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:15.428 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:15.428 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:15.428 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:15.429 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:15.429 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:15.429 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:15.429 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:15.430 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:15.430 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:15.430 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:15.430 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:15.431 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:15.431 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:15.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:15.431 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:15.432 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:15.432 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:15.432 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:15.432 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:15.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.733637_169.1 2021-09-06T11:59:20.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:20.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:20.436 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:20.485 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:20.485 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:20.563 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:20.564 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:20.564 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:20.564 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:20.564 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:20.565 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:20.565 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:20.565 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:20.565 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:20.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:20.566 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:20.566 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:20.566 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:20.567 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:20.567 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:20.567 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:20.568 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:20.568 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:20.568 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:20.568 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:20.568 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:20.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:20.569 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:20.569 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:20.569 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:20.570 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:20.570 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:20.570 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:20.570 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:20.571 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:20.571 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:20.571 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:20.571 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:20.572 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:20.572 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:20.572 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:20.572 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:20.573 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:59:20.573 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:20.573 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:20.573 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:20.573 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:59:20.574 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:20.574 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:20.574 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:20.574 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:20.575 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:20.575 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:20.575 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:20.575 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:20.576 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:20.576 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:20.576 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:20.576 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:20.576 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:20.577 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:20.577 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:20.577 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:20.577 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:20.578 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:20.578 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:20.578 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:20.578 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:20.579 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:20.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.738114_171.1 2021-09-06T11:59:25.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:25.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:25.583 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:25.664 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:25.664 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:25.961 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:25.962 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:25.962 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:25.962 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:25.963 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:25.963 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:25.963 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:25.963 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:25.964 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:25.964 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:25.964 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:25.964 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:25.965 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:25.965 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:25.965 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:25.965 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:25.966 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:25.966 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:25.966 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:25.966 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:25.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:25.967 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:25.967 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:25.967 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:25.967 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:25.968 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:25.968 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:25.968 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:25.968 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:25.969 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:25.969 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:25.969 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:25.969 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:25.969 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:25.970 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:25.970 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:25.970 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:25.970 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.738114_171.1", 2021-09-06T11:59:25.971 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:25.971 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:25.971 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:25.971 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.738114Z", 2021-09-06T11:59:25.972 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:25.972 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:25.972 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:25.972 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:25.972 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:25.973 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:25.973 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:25.973 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:25.973 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:25.974 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:25.974 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:25.974 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:25.974 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:25.975 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:25.975 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:25.975 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:25.975 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:25.975 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:25.976 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:25.976 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:25.976 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:25.976 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:25.978 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.738114_171.1 2021-09-06T11:59:30.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:30.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:30.981 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:31.031 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:31.032 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:31.098 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:31.099 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:31.099 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:31.099 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:31.100 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:31.100 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:31.100 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:31.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:31.101 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:31.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:31.101 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:31.101 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:31.102 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:31.102 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:31.102 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:31.102 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:31.103 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:31.103 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:31.103 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:31.103 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:31.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:31.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:31.104 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:31.104 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:31.104 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:31.105 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:31.105 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:31.105 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:31.105 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:31.105 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:31.106 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:31.106 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:31.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:31.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:31.107 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:31.107 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:31.107 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:31.107 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.565140_164.1", 2021-09-06T11:59:31.107 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:31.108 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:31.108 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:31.108 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.565140Z", 2021-09-06T11:59:31.108 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:31.109 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:31.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:31.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:31.109 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:31.110 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:31.110 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:31.110 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:31.110 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:31.110 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:31.111 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:31.111 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:31.111 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:31.111 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:31.112 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:31.112 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:31.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:31.112 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:31.112 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:31.113 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:31.113 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:31.113 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:31.114 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.565140_164.1 2021-09-06T11:59:36.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:36.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:36.118 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:36.169 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:36.169 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:36.783 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:36.783 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:36.784 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:36.784 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:36.784 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:36.784 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:36.784 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:36.785 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:36.785 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:36.785 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:36.785 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:36.786 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:36.786 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:36.786 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:36.786 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:36.787 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:36.787 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:36.787 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:36.787 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:36.788 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:36.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:36.788 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:36.788 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:36.788 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:36.789 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:36.789 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:36.789 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:36.789 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:36.790 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:36.790 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:36.790 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:36.790 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:36.791 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:36.791 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:36.791 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:36.791 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:36.792 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:36.792 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.741261_172.1", 2021-09-06T11:59:36.792 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:36.792 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:36.792 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:36.793 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.741261Z", 2021-09-06T11:59:36.793 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:36.793 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:36.793 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:36.794 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:36.794 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:36.794 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:36.794 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:36.794 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:36.795 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:36.795 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:36.795 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:36.795 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:36.796 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:36.796 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:36.796 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:36.796 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:36.797 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:36.797 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:36.797 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:36.797 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:36.797 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:36.798 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:36.799 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.741261_172.1 2021-09-06T11:59:41.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:41.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:41.802 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:41.850 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:41.850 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:41.927 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:41.927 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:41.927 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:41.928 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:41.928 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:41.928 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:41.928 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:41.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:41.929 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:41.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:41.929 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:41.930 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:41.930 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:41.930 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:41.930 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:41.931 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:41.931 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:41.931 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:41.931 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:41.931 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:41.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:41.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:41.932 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:41.932 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:41.933 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:41.933 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:41.933 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:41.933 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:41.933 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:41.934 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:41.934 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:41.934 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:41.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:41.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:41.935 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:41.935 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:41.935 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:41.936 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:59:41.936 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:41.936 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:41.936 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:41.936 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:59:41.937 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:41.937 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:41.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:41.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:41.938 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:41.938 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:41.938 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:41.938 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:41.939 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:41.939 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:41.939 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:41.939 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:41.939 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:41.940 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:41.940 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:41.940 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:41.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:41.941 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:41.941 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:41.941 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:41.941 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:41.941 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:41.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T11:59:46.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:46.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:46.945 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:46.988 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:46.989 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:47.061 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:47.062 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:47.062 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:47.062 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:47.063 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:47.063 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:47.063 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:47.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:47.064 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:47.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:47.064 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:47.064 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:47.065 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:47.065 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:47.065 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:47.065 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:47.066 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:47.066 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:47.066 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:47.066 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:47.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:47.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:47.067 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:47.068 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:47.068 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:47.068 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:47.068 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:47.069 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:47.069 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:47.069 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:47.069 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:47.070 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:47.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:47.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:47.070 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:47.071 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:47.071 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:47.071 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.543029_160.1", 2021-09-06T11:59:47.071 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:47.072 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:47.072 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:47.072 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.543029Z", 2021-09-06T11:59:47.073 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:47.073 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:47.073 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:47.073 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:47.074 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:47.074 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:47.074 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:47.074 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:47.074 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:47.075 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:47.075 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:47.075 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:47.075 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:47.076 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:47.076 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:47.076 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:47.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:47.077 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:47.077 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:47.077 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:47.078 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:47.078 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:47.079 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.543029_160.1 2021-09-06T11:59:52.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:52.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:52.125 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:52.184 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:52.184 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:52.624 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:52.624 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:52.625 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:52.625 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:52.625 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:52.625 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:52.626 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:52.626 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:52.626 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:52.626 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:52.627 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:52.627 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:52.627 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:52.627 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:52.627 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:52.628 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:52.628 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:52.628 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:52.628 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:52.629 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:52.629 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:52.629 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:52.629 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:52.630 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:52.630 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:52.630 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:52.630 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:52.630 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:52.631 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:52.631 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:52.631 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:52.631 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:52.632 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:52.632 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:52.632 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:52.632 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:52.633 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:52.633 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T11:59:52.633 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:52.633 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:52.633 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:52.634 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T11:59:52.634 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:52.634 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:52.634 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:52.635 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:52.635 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:52.635 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:52.635 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:52.636 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:52.636 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:52.636 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:52.636 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:52.636 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:52.637 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:52.637 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:52.637 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:52.637 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:52.638 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:52.638 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:52.638 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:52.639 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:52.639 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:52.639 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:52.640 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T11:59:57.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:57.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T11:59:57.643 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T11:59:57.688 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T11:59:57.688 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T11:59:57.755 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T11:59:57.755 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T11:59:57.755 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T11:59:57.756 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T11:59:57.756 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T11:59:57.756 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T11:59:57.756 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:57.756 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:57.757 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T11:59:57.757 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:57.757 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T11:59:57.757 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:57.758 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:57.758 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T11:59:57.758 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:57.758 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T11:59:57.758 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:57.759 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T11:59:57.759 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:57.759 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:57.759 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:57.759 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:57.760 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T11:59:57.760 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:57.760 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:57.760 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T11:59:57.761 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:57.761 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:57.761 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:57.761 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T11:59:57.761 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:57.762 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:57.762 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:57.762 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:57.762 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T11:59:57.763 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:57.763 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:57.763 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.539819_159.1", 2021-09-06T11:59:57.763 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:57.764 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:57.764 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:57.764 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.539819Z", 2021-09-06T11:59:57.764 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:57.765 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:57.765 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:57.765 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T11:59:57.765 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T11:59:57.765 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T11:59:57.766 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T11:59:57.766 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T11:59:57.766 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T11:59:57.766 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T11:59:57.767 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T11:59:57.767 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T11:59:57.767 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T11:59:57.767 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:57.768 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:57.768 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T11:59:57.768 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T11:59:57.768 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T11:59:57.769 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T11:59:57.769 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T11:59:57.769 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T11:59:57.769 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T11:59:57.770 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.539819_159.1 2021-09-06T12:00:02.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:02.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:02.773 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:02.856 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:02.856 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:03.298 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:03.299 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:03.299 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:03.299 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:03.300 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:03.300 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:03.300 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:03.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:03.301 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:03.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:03.301 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:03.301 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:03.302 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:03.302 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:03.303 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:03.303 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:03.303 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:03.304 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:03.304 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:03.304 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:03.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:03.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:03.305 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:03.305 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:03.305 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:03.305 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:03.306 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:03.306 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:03.306 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:03.306 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:03.307 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:03.307 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:03.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:03.307 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:03.308 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:03.308 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:03.308 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:03.308 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T12:00:03.309 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:03.309 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:03.309 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:03.309 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T12:00:03.310 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:03.310 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:03.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:03.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:03.311 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:03.311 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:03.311 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:03.311 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:03.312 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:03.312 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:03.312 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:03.312 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:03.313 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:03.313 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:03.313 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:03.313 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:03.314 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:03.314 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:03.314 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:03.314 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:03.314 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:03.315 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:03.316 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.489232_154.1 2021-09-06T12:00:08.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:08.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:08.320 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:08.369 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:08.369 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:08.438 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:08.439 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:08.439 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:08.439 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:08.439 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:08.440 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:08.440 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:08.440 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:08.440 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:08.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:08.441 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:08.441 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:08.441 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:08.442 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:08.442 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:08.442 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:08.443 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:08.443 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:08.443 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:08.443 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:08.444 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:08.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:08.444 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:08.444 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:08.445 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:08.445 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:08.445 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:08.445 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:08.446 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:08.446 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:08.446 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:08.446 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:08.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:08.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:08.447 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:08.447 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:08.448 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:08.448 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T12:00:08.448 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:08.449 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:08.449 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:08.449 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T12:00:08.449 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:08.450 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:08.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:08.450 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:08.450 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:08.451 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:08.451 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:08.451 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:08.451 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:08.452 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:08.452 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:08.452 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:08.452 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:08.453 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:08.453 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:08.453 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:08.453 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:08.454 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:08.454 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:08.454 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:08.454 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:08.455 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:08.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.544174_161.1 2021-09-06T12:00:13.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:13.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:13.459 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:13.501 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:13.502 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:13.566 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:13.566 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:13.566 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:13.566 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:13.567 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:13.567 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:13.567 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:13.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:13.568 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:13.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:13.568 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:13.568 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:13.569 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:13.569 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:13.569 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:13.569 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:13.570 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:13.570 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:13.570 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:13.570 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:13.571 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:13.571 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:13.571 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:13.571 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:13.572 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:13.572 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:13.572 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:13.572 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:13.572 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:13.573 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:13.573 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:13.573 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:13.573 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:13.574 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:13.574 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:13.574 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:13.574 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:13.575 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T12:00:13.575 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:13.575 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:13.575 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:13.575 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T12:00:13.576 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:13.576 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:13.576 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:13.576 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:13.577 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:13.577 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:13.577 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:13.577 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:13.577 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:13.578 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:13.578 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:13.578 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:13.578 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:13.579 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:13.579 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:13.579 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:13.579 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:13.580 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:13.580 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:13.580 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:13.580 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:13.581 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:13.582 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.507089_156.1 2021-09-06T12:00:18.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:18.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:18.585 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:18.665 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:18.665 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:18.867 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:18.867 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:18.867 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:18.867 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:18.868 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:18.868 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:18.868 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:18.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:18.869 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:18.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:18.869 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:18.869 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:18.870 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:18.870 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:18.870 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:18.870 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:18.871 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:18.871 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:18.871 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:18.871 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:18.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:18.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:18.872 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:18.872 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:18.872 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:18.873 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:18.873 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:18.873 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:18.873 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:18.873 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:18.874 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:18.874 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:18.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:18.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:18.875 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:18.875 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:18.875 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:18.875 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.798549_179.1", 2021-09-06T12:00:18.876 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:18.876 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:18.876 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:18.876 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.798549Z", 2021-09-06T12:00:18.877 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:18.877 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:18.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:18.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:18.878 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:18.878 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:18.878 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:18.878 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:18.879 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:18.879 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:18.879 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:18.879 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:18.880 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:18.880 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:18.880 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:18.880 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:18.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:18.881 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:18.881 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:18.881 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:18.881 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:18.882 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:18.883 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.798549_179.1 2021-09-06T12:00:23.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:23.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:23.886 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:23.952 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:23.952 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:24.077 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:24.078 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:24.078 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:24.078 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:24.079 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:24.079 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:24.079 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:24.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:24.080 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:24.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:24.080 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:24.080 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:24.081 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:24.081 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:24.081 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:24.081 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:24.081 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:24.082 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:24.082 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:24.082 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:24.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:24.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:24.083 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:24.083 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:24.083 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:24.083 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:24.084 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:24.084 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:24.084 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:24.084 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:24.084 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:24.085 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:24.085 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:24.085 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:24.085 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:24.085 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:24.086 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:24.086 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T12:00:24.086 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:24.086 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:24.087 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:24.087 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T12:00:24.087 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:24.087 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:24.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:24.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:24.088 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:24.088 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:24.088 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:24.088 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:24.089 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:24.089 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:24.089 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:24.089 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:24.089 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:24.090 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:24.090 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:24.090 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:24.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:24.091 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:24.091 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:24.091 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:24.091 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:24.091 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:24.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.489232_154.1 2021-09-06T12:00:29.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:29.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:29.095 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:29.155 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:29.155 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:29.816 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:29.817 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:29.817 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:29.817 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:29.817 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:29.818 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:29.818 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:29.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:29.818 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:29.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:29.819 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:29.819 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:29.819 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:29.820 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:29.820 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:29.820 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:29.820 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:29.820 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:29.821 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:29.821 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:29.821 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:29.821 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:29.822 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:29.822 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:29.822 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:29.822 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:29.823 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:29.823 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:29.823 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:29.823 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:29.824 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:29.824 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:29.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:29.824 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:29.824 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:29.825 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:29.825 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:29.825 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.676443_168.1", 2021-09-06T12:00:29.826 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:29.826 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:29.826 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:29.826 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.676443Z", 2021-09-06T12:00:29.827 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:29.827 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:29.827 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:29.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:29.828 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:29.828 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:29.828 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:29.829 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:29.829 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:29.829 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:29.829 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:29.830 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:29.830 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:29.830 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:29.830 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:29.831 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:29.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:29.831 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:29.831 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:29.832 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:29.832 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:29.832 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:29.833 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.676443_168.1 2021-09-06T12:00:34.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:34.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:34.837 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:34.885 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:34.885 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:34.957 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:34.957 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:34.958 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:34.958 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:34.958 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:34.958 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:34.959 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:34.959 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:34.959 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:34.959 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:34.959 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:34.960 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:34.960 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:34.960 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:34.960 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:34.961 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:34.961 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:34.961 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:34.961 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:34.961 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:34.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:34.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:34.962 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:34.962 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:34.963 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:34.963 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:34.963 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:34.963 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:34.964 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:34.964 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:34.964 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:34.964 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:34.964 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:34.965 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:34.965 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:34.965 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:34.965 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:34.965 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T12:00:34.966 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:34.966 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:34.966 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:34.966 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T12:00:34.967 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:34.967 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:34.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:34.967 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:34.967 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:34.968 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:34.968 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:34.968 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:34.968 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:34.969 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:34.969 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:34.969 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:34.969 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:34.969 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:34.970 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:34.970 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:34.970 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:34.970 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:34.971 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:34.971 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:34.971 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:34.971 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:34.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.544174_161.1 2021-09-06T12:00:39.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:39.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:39.976 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:40.024 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:40.024 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:40.101 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:40.102 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:40.102 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:40.102 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:40.102 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:40.103 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:40.103 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:40.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:40.104 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:40.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:40.104 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:40.104 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:40.104 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:40.105 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:40.105 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:40.105 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:40.106 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:40.106 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:40.107 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:40.107 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:40.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:40.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:40.108 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:40.109 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:40.109 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:40.109 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:40.110 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:40.110 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:40.110 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:40.110 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:40.111 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:40.111 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:40.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:40.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:40.112 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:40.113 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:40.113 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:40.114 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T12:00:40.114 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:40.114 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:40.114 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:40.115 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T12:00:40.115 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:40.115 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:40.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:40.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:40.116 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:40.117 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:40.117 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:40.117 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:40.118 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:40.118 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:40.118 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:40.118 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:40.118 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:40.119 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:40.119 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:40.119 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:40.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:40.120 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:40.120 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:40.120 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:40.120 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:40.121 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:40.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.544174_161.1 2021-09-06T12:00:45.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:45.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:45.125 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:45.202 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:45.203 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:45.614 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:45.614 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:45.615 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:45.616 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:45.617 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:45.618 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:45.619 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:45.620 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:45.621 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:45.622 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T12:00:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:45.623 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T12:00:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:45.624 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:45.625 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:45.626 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:45.627 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:45.627 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:45.627 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:45.627 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:45.628 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:45.628 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:45.628 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:45.628 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:45.628 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:45.629 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:45.629 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:45.629 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:45.630 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.544174_161.1 2021-09-06T12:00:50.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:50.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:50.633 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:50.730 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:50.731 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:51.022 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:51.023 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:51.023 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:51.023 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:51.024 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:51.024 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:51.024 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:51.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:51.024 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:51.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:51.025 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:51.025 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:51.025 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:51.026 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:51.026 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:51.026 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:51.026 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:51.026 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:51.027 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:51.027 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:51.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:51.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:51.028 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:51.028 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:51.028 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:51.028 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:51.028 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:51.029 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:51.029 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:51.029 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:51.029 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:51.030 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:51.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:51.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:51.030 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:51.030 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:51.031 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:51.031 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T12:00:51.031 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:51.031 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:51.032 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:51.032 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T12:00:51.032 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:51.032 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:51.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:51.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:51.033 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:51.033 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:51.033 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:51.034 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:51.034 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:51.034 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:51.034 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:51.034 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:51.035 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:51.035 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:51.035 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:51.035 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:51.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:51.036 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:51.036 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:51.036 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:51.036 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:51.037 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:51.038 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.544174_161.1 2021-09-06T12:00:56.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:56.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:00:56.041 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:00:56.111 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:00:56.111 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:00:56.826 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:00:56.826 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:00:56.826 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:00:56.827 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:00:56.827 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:00:56.827 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:00:56.828 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:56.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:56.828 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:00:56.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:56.829 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:00:56.829 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:56.829 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:56.829 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:00:56.830 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:56.830 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:00:56.830 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:56.830 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:00:56.831 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:56.831 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:56.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:56.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:56.832 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:00:56.832 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:56.832 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:56.832 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:00:56.833 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:56.833 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:56.833 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:56.833 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:00:56.834 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:56.834 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:56.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:56.834 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:56.835 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:00:56.835 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:56.835 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:56.835 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.611335_165.1", 2021-09-06T12:00:56.836 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:56.836 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:56.836 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:56.836 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.611335Z", 2021-09-06T12:00:56.837 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:56.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:56.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:56.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:00:56.838 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:00:56.838 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:00:56.838 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:00:56.838 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:00:56.839 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:00:56.839 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:00:56.839 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:00:56.839 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:00:56.839 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:00:56.840 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:56.840 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:56.840 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:00:56.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:00:56.841 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:00:56.841 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:00:56.841 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:00:56.841 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:00:56.842 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:00:56.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.611335_165.1 2021-09-06T12:01:01.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:01.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:01.846 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:01.922 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:01.922 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:02.589 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:02.590 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:02.590 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:02.590 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:02.590 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:02.591 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:02.591 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:02.591 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:02.591 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:02.592 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:02.592 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:02.592 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:02.592 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:02.593 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:02.593 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:02.593 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:02.593 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:02.594 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:02.594 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:02.594 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:02.594 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:02.594 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:02.595 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:02.595 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:02.595 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:02.595 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:02.596 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:02.596 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:02.596 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:02.596 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:02.597 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:02.597 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:02.597 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:02.597 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:02.597 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:02.598 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:02.598 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:02.598 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T12:01:02.598 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:02.599 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:02.599 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:02.599 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T12:01:02.599 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:02.600 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:02.600 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:02.600 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:02.600 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:02.600 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:02.601 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:02.601 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:02.601 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:02.601 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:02.602 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:02.602 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:02.602 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:02.602 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:02.603 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:02.603 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:02.603 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:02.603 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:02.604 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:02.604 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:02.604 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:02.604 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:02.605 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.503662_155.1 2021-09-06T12:01:07.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:07.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:07.608 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:07.657 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:07.657 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:07.731 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:07.731 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:07.731 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:07.731 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:07.732 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:07.732 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:07.732 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:07.732 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:07.733 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:07.733 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:07.733 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:07.733 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:07.733 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:07.734 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:07.734 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:07.734 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:07.734 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:07.735 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:07.735 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:07.735 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:07.735 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:07.735 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:07.736 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:07.736 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:07.736 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:07.736 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:07.737 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:07.737 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:07.737 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:07.737 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:07.737 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:07.738 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:07.738 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:07.738 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:07.738 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:07.738 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:07.739 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:07.739 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.544174_161.1", 2021-09-06T12:01:07.739 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:07.739 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:07.739 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:07.740 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.544174Z", 2021-09-06T12:01:07.740 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:07.740 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:07.740 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:07.740 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:07.741 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:07.741 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:07.741 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:07.741 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:07.742 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:07.742 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:07.742 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:07.742 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:07.742 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:07.743 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:07.743 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:07.743 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:07.743 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:07.744 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:07.744 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:07.744 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:07.744 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:07.744 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:07.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.544174_161.1 2021-09-06T12:01:12.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:12.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:12.749 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:12.791 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:12.791 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:12.857 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:12.857 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:12.857 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:12.857 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:12.858 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:12.858 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:12.858 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:12.858 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:12.858 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:12.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:12.859 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:12.859 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:12.859 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:12.859 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:12.860 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:12.860 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:12.860 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:12.860 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:12.861 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:12.861 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:12.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:12.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:12.861 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:12.862 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:12.862 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:12.862 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:12.862 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:12.862 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:12.863 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:12.863 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:12.863 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:12.863 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:12.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:12.864 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:12.864 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:12.864 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:12.864 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:12.865 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T12:01:12.865 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:12.865 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:12.865 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:12.865 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T12:01:12.866 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:12.866 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:12.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:12.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:12.867 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:12.867 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:12.867 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:12.867 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:12.868 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:12.868 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:12.868 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:12.868 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:12.869 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:12.869 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:12.869 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:12.869 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:12.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:12.870 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:12.870 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:12.870 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:12.870 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:12.871 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:12.872 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.489232_154.1 2021-09-06T12:01:17.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:17.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:17.875 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:17.918 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:17.919 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:18.197 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:18.197 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:18.198 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:18.198 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:18.198 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:18.198 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:18.199 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:18.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:18.199 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:18.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:18.200 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:18.200 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:18.200 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:18.200 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:18.201 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:18.201 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:18.201 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:18.201 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:18.201 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:18.202 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:18.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:18.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:18.202 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:18.203 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:18.203 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:18.203 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:18.203 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:18.203 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:18.204 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:18.204 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:18.204 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:18.204 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:18.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:18.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:18.205 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:18.205 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:18.206 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:18.206 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T12:01:18.206 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:18.206 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:18.206 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:18.207 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T12:01:18.207 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:18.207 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:18.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:18.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:18.208 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:18.208 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:18.208 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:18.209 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:18.209 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:18.209 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:18.209 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:18.209 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:18.210 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:18.210 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:18.210 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:18.210 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:18.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:18.211 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:18.211 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:18.211 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:18.211 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:18.212 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:18.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.507089_156.1 2021-09-06T12:01:23.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:23.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:23.216 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:23.287 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:23.288 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:23.637 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:23.638 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:23.638 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:23.638 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:23.639 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:23.639 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:23.639 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:23.639 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:23.639 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:23.640 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:23.640 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:23.640 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:23.640 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:23.641 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:23.641 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:23.641 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:23.641 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:23.642 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:23.642 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:23.642 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:23.642 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:23.643 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:23.643 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:23.643 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:23.643 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:23.644 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:23.644 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:23.644 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:23.644 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:23.644 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:23.645 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:23.645 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:23.645 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:23.645 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:23.646 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:23.646 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:23.646 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:23.646 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T12:01:23.646 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:23.647 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:23.647 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:23.647 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T12:01:23.647 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:23.647 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:23.648 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:23.648 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:23.648 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:23.648 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:23.649 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:23.649 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:23.649 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:23.649 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:23.649 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:23.650 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:23.650 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:23.650 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:23.650 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:23.651 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:23.651 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:23.651 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:23.651 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:23.651 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:23.652 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:23.652 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:23.653 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T12:01:28.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:28.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:28.656 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:28.701 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:28.701 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:28.773 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:28.774 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:28.774 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:28.774 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:28.774 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:28.775 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:28.775 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:28.775 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:28.775 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:28.776 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:28.776 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:28.776 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:28.776 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:28.777 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:28.777 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:28.777 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:28.777 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:28.778 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:28.778 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:28.778 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:28.778 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:28.779 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:28.779 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:28.779 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:28.779 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:28.779 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:28.780 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:28.780 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:28.780 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:28.780 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:28.781 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:28.781 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:28.781 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:28.781 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:28.782 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:28.782 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:28.782 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:28.782 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.489232_154.1", 2021-09-06T12:01:28.783 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:28.783 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:28.783 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:28.783 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.489232Z", 2021-09-06T12:01:28.784 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:28.784 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:28.784 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:28.784 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:28.784 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:28.785 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:28.785 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:28.785 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:28.785 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:28.786 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:28.786 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:28.786 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:28.786 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:28.786 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:28.787 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:28.787 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:28.787 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:28.787 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:28.788 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:28.788 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:28.788 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:28.788 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:28.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.489232_154.1 2021-09-06T12:01:33.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:33.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:33.792 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:33.835 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:33.836 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:33.904 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:33.905 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:33.905 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:33.905 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:33.905 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:33.906 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:33.906 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:33.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:33.906 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:33.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:33.907 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:33.907 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:33.907 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:33.907 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:33.908 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:33.908 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:33.908 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:33.908 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:33.909 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:33.909 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:33.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:33.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:33.910 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:33.910 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:33.910 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:33.910 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:33.911 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:33.911 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:33.911 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:33.911 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:33.911 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:33.912 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:33.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:33.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:33.912 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:33.913 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:33.913 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:33.913 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T12:01:33.913 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:33.913 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:33.914 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:33.914 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T12:01:33.914 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:33.914 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:33.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:33.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:33.915 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:33.915 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:33.915 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:33.916 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:33.916 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:33.916 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:33.916 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:33.917 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:33.917 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:33.917 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:33.917 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:33.917 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:33.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:33.918 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:33.918 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:33.918 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:33.919 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:33.919 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:33.920 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.507089_156.1 2021-09-06T12:01:38.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:38.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:38.923 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:39.000 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:39.000 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:39.280 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:39.281 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:39.281 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:39.281 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:39.282 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:39.282 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:39.282 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:39.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:39.283 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:39.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:39.283 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:39.283 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:39.283 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:39.284 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:39.284 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:39.284 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:39.284 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:39.285 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:39.285 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:39.285 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:39.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:39.286 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:39.286 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:39.286 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:39.286 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:39.286 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:39.287 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:39.287 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:39.287 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:39.287 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:39.287 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:39.288 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:39.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:39.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:39.288 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:39.289 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:39.289 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:39.289 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T12:01:39.289 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:39.289 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:39.290 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:39.290 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T12:01:39.290 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:39.290 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:39.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:39.291 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:39.291 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:39.291 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:39.291 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:39.291 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:39.292 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:39.292 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:39.292 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:39.292 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:39.293 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:39.293 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:39.293 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:39.293 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:39.293 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:39.294 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:39.294 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:39.294 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:39.294 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:39.295 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:39.296 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.503662_155.1 2021-09-06T12:01:44.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:44.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:44.299 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:44.346 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:44.346 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:44.671 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:44.671 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:44.672 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:44.672 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:44.672 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:44.672 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:44.673 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:44.673 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:44.673 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:44.673 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:44.674 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:44.674 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:44.674 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:44.674 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:44.674 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:44.675 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:44.675 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:44.675 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:44.675 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:44.676 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:44.676 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:44.676 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:44.676 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:44.677 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:44.677 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:44.677 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:44.677 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:44.677 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:44.678 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:44.678 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:44.678 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:44.678 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:44.679 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:44.679 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:44.679 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:44.679 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:44.680 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:44.680 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.539819_159.1", 2021-09-06T12:01:44.680 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:44.680 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:44.681 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:44.681 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.539819Z", 2021-09-06T12:01:44.681 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:44.681 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:44.681 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:44.682 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:44.682 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:44.682 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:44.682 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:44.682 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:44.683 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:44.683 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:44.683 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:44.683 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:44.684 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:44.684 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:44.684 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:44.684 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:44.684 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:44.685 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:44.685 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:44.685 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:44.685 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:44.686 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:44.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.539819_159.1 2021-09-06T12:01:49.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:49.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:49.690 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:49.770 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:49.770 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:50.184 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:50.184 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:50.184 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:50.185 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:50.185 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:50.185 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:50.185 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:50.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:50.186 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:50.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:50.186 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:50.186 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:50.187 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:50.187 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:50.187 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:50.188 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:50.188 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:50.188 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:50.189 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:50.189 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:50.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:50.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:50.190 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:50.190 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:50.190 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:50.190 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:50.191 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:50.191 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:50.191 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:50.191 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:50.192 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:50.192 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:50.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:50.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:50.192 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:50.193 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:50.193 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:50.193 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.797272_178.1", 2021-09-06T12:01:50.193 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:50.194 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:50.194 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:50.194 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.797272Z", 2021-09-06T12:01:50.194 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:50.195 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:50.196 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:50.196 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:50.196 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:50.197 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:50.197 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:50.197 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:50.197 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:50.197 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:50.198 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:50.198 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:50.198 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:50.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:50.199 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:50.199 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:50.199 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:50.199 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:50.200 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:50.201 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.797272_178.1 2021-09-06T12:01:55.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:55.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:01:55.204 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:01:55.250 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:01:55.250 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:01:55.322 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:01:55.322 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:01:55.322 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:01:55.322 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:01:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:01:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:01:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:55.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:55.324 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:01:55.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:55.324 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:01:55.324 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:55.325 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:55.325 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:01:55.325 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:55.325 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:01:55.325 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:55.326 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:01:55.326 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:55.326 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:55.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:55.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:55.327 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:01:55.327 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:55.327 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:55.328 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:01:55.328 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:55.328 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:55.328 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:55.328 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:01:55.329 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:55.329 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:55.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:55.329 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:55.330 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:01:55.330 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:55.330 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:55.330 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.798549_179.1", 2021-09-06T12:01:55.331 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:55.331 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:55.331 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:55.331 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.798549Z", 2021-09-06T12:01:55.331 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:55.332 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:55.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:55.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:01:55.332 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:01:55.333 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:01:55.333 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:01:55.333 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:01:55.333 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:01:55.334 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:01:55.334 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:01:55.334 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:01:55.334 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:01:55.334 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:55.335 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:55.335 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:01:55.335 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:01:55.335 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:01:55.336 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:01:55.336 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:01:55.336 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:01:55.336 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:01:55.337 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.798549_179.1 2021-09-06T12:02:00.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:00.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:00.340 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:02:00.389 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:00.390 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:00.464 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:02:00.464 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:02:00.465 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:02:00.465 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:02:00.465 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:02:00.465 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:00.466 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:00.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:00.466 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:02:00.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:00.467 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:02:00.467 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:00.467 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:00.467 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:02:00.468 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:00.468 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:02:00.468 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:00.468 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:02:00.468 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:00.469 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:00.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:00.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:00.469 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:02:00.470 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:00.470 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:00.470 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:02:00.470 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:00.471 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:00.471 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:00.471 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:02:00.471 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:00.471 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:00.472 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:00.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:00.472 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:02:00.472 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:00.473 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:00.473 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.511634_158.1", 2021-09-06T12:02:00.473 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:00.473 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:00.473 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:00.474 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.511634Z", 2021-09-06T12:02:00.474 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:00.474 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:00.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:00.475 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:00.475 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:02:00.475 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:00.475 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:00.476 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:02:00.476 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:00.476 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:00.476 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:00.476 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:02:00.477 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:00.477 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:00.477 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:00.477 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:02:00.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:00.478 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:02:00.478 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:02:00.478 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:02:00.479 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:00.479 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:02:00.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.511634_158.1 2021-09-06T12:02:05.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:05.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:05.483 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:02:05.566 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:05.567 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:05.820 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:02:05.821 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:02:05.821 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:02:05.821 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:02:05.821 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:02:05.822 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:05.822 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:05.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:05.822 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:02:05.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:05.823 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:02:05.823 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:05.823 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:05.824 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:02:05.824 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:05.824 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:02:05.824 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:05.824 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:02:05.825 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:05.825 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:05.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:05.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:05.826 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:02:05.826 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:05.826 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:05.826 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:02:05.826 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:05.827 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:05.827 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:05.827 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:02:05.827 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:05.828 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:05.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:05.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:05.828 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:02:05.828 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:05.829 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:05.829 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T12:02:05.829 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:05.829 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:05.830 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:05.830 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T12:02:05.830 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:05.830 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:05.830 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:05.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:05.831 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:02:05.831 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:05.831 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:05.832 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:02:05.832 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:05.832 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:05.832 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:05.832 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:02:05.833 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:05.833 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:05.833 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:05.833 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:02:05.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:05.834 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:02:05.834 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:02:05.834 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:02:05.834 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:05.835 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:02:05.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.503662_155.1 2021-09-06T12:02:10.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:10.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:10.839 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:02:10.892 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:10.892 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:10.973 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:02:10.973 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:02:10.973 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:02:10.973 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:02:10.974 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:02:10.974 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:10.974 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:10.974 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:10.975 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:02:10.975 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:10.975 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:02:10.975 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:10.976 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:10.976 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:02:10.976 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:10.976 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:02:10.977 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:10.977 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:02:10.977 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:10.977 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:10.978 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:10.978 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:10.978 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:02:10.978 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:10.979 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:10.979 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:02:10.979 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:10.979 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:10.980 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:10.980 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:02:10.980 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:10.980 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:10.981 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:10.981 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:10.981 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:02:10.981 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:10.981 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:10.982 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.676443_168.1", 2021-09-06T12:02:10.982 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:10.982 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:10.982 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:10.983 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.676443Z", 2021-09-06T12:02:10.983 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:10.983 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:10.983 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:10.984 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:10.984 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:02:10.984 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:10.984 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:10.985 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:02:10.985 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:10.985 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:10.985 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:10.985 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:02:10.986 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:10.986 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:10.986 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:10.986 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:02:10.987 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:10.987 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:02:10.987 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:02:10.987 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:02:10.988 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:10.988 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:02:10.989 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.676443_168.1 2021-09-06T12:02:15.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:15.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:15.992 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:02:16.042 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:16.043 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:16.138 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:02:16.138 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:02:16.138 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:02:16.139 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:02:16.139 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:02:16.139 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:16.139 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:16.140 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:16.140 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:02:16.140 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:16.140 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:02:16.141 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:16.141 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:16.141 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:02:16.141 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:16.141 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:02:16.142 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:16.142 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:02:16.142 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:16.142 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:16.143 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:16.143 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:16.143 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:02:16.143 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:16.144 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:16.144 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:02:16.144 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:16.144 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:16.144 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:16.145 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:02:16.145 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:16.145 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:16.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:16.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:16.146 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:02:16.146 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:16.146 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:16.147 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.503662_155.1", 2021-09-06T12:02:16.147 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:16.147 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:16.147 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:16.148 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.503662Z", 2021-09-06T12:02:16.148 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:16.148 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:16.148 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:16.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:16.149 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:02:16.149 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:16.149 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:16.149 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:02:16.150 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:16.150 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:16.150 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:16.150 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:02:16.151 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:16.151 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:16.151 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:16.151 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:02:16.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:16.152 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:02:16.152 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:02:16.152 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:02:16.152 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:16.153 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:02:16.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.503662_155.1 2021-09-06T12:02:21.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:21.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:21.157 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:02:21.200 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:21.201 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:21.273 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:02:21.273 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:02:21.274 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:02:21.274 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:02:21.274 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:02:21.275 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:21.275 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:21.275 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:21.275 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:02:21.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:21.276 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:02:21.276 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:21.276 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:21.277 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:02:21.277 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:21.277 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:02:21.277 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:21.277 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:02:21.278 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:21.278 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:21.278 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:21.278 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:21.279 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:02:21.279 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:21.279 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:21.279 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:02:21.279 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:21.280 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:21.280 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:21.280 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:02:21.280 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:21.281 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:21.281 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:21.281 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:21.281 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:02:21.281 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:21.282 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:21.282 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.561874_163.1", 2021-09-06T12:02:21.282 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:21.282 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:21.283 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:21.283 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.561874Z", 2021-09-06T12:02:21.283 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:21.283 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:21.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:21.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:21.284 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:02:21.284 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:21.284 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:21.284 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:02:21.285 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:21.285 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:21.285 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:21.285 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:02:21.286 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:21.286 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:21.286 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:21.286 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:02:21.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:21.287 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:02:21.287 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:02:21.287 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:02:21.287 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:21.288 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:02:21.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.561874_163.1 2021-09-06T12:02:26.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:26.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:26.292 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:02:26.375 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:26.376 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:26.661 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:02:26.661 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:02:26.661 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:02:26.662 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:02:26.662 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:02:26.662 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:26.662 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:26.663 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:26.663 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:02:26.663 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:26.663 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:02:26.664 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:26.664 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:26.664 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:02:26.664 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:26.664 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:02:26.665 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:26.665 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:02:26.665 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:26.665 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:26.666 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:26.666 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:26.666 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:02:26.666 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:26.667 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:26.667 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:02:26.667 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:26.667 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:26.667 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:26.668 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:02:26.668 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:26.668 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:26.668 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:26.669 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:26.669 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:02:26.669 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:26.669 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:26.669 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.539819_159.1", 2021-09-06T12:02:26.670 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:26.670 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:26.670 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:26.670 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.539819Z", 2021-09-06T12:02:26.671 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:26.671 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:26.671 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:26.671 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:26.672 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:02:26.672 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:26.672 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:26.672 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:02:26.672 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:26.673 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:26.673 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:26.673 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:02:26.673 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:26.674 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:26.674 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:26.674 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:02:26.674 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:26.674 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:02:26.675 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:02:26.675 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:02:26.675 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:26.675 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:02:26.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.539819_159.1 2021-09-06T12:02:31.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:31.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:02:31.681 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:02:31.733 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:31.733 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:32.019 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-09-06T12:02:32.019 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2021-09-06T12:02:32.019 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2021-09-06T12:02:32.020 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2021-09-06T12:02:32.020 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2021-09-06T12:02:32.020 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:32.021 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:32.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:32.021 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2021-09-06T12:02:32.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:32.022 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2021-09-06T12:02:32.022 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:32.022 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:32.022 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927514.716931_25.1", 2021-09-06T12:02:32.023 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:32.023 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2021-09-06T12:02:32.023 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:32.023 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:25:14.716931Z", 2021-09-06T12:02:32.024 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:32.024 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:32.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:32.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:32.025 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2021-09-06T12:02:32.025 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:32.025 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:32.025 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2021-09-06T12:02:32.025 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:32.026 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:32.026 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:32.026 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2021-09-06T12:02:32.026 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:32.027 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:32.027 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:32.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:32.027 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2021-09-06T12:02:32.028 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:32.028 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:32.028 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630928091.507089_156.1", 2021-09-06T12:02:32.029 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:32.029 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:32.029 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:32.029 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:34:51.507089Z", 2021-09-06T12:02:32.029 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:32.030 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:32.030 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:32.030 INFO:teuthology.orchestra.run.smithi139.stdout: { 2021-09-06T12:02:32.030 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2021-09-06T12:02:32.031 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2021-09-06T12:02:32.031 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2021-09-06T12:02:32.031 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1630927659.337096_88.1", 2021-09-06T12:02:32.031 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2021-09-06T12:02:32.032 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2021-09-06T12:02:32.032 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2021-09-06T12:02:32.032 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2021-09-06T11:27:39.337096Z", 2021-09-06T12:02:32.032 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2021-09-06T12:02:32.033 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:32.033 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:32.033 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2021-09-06T12:02:32.033 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2021-09-06T12:02:32.033 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2021-09-06T12:02:32.034 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2021-09-06T12:02:32.034 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2021-09-06T12:02:32.034 INFO:teuthology.orchestra.run.smithi139.stdout: } 2021-09-06T12:02:32.034 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-09-06T12:02:32.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630929121.899863_275.1 target=1_1630928091.507089_156.1 2021-09-06T12:02:37.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2021-09-06T12:02:37.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:37.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:37.040 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:02:37.279 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:37.279 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:37.383 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T12:02:37.384 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "208703f9-6eb4-4701-a7f2-dfe84295cfa0", 2021-09-06T12:02:37.384 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "remove_zg", 2021-09-06T12:02:37.384 INFO:teuthology.orchestra.run.smithi094.stdout: "api_name": "remove_zg", 2021-09-06T12:02:37.384 INFO:teuthology.orchestra.run.smithi094.stdout: "is_master": "false", 2021-09-06T12:02:37.385 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [], 2021-09-06T12:02:37.385 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames": [], 2021-09-06T12:02:37.385 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames_s3website": [], 2021-09-06T12:02:37.385 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "", 2021-09-06T12:02:37.386 INFO:teuthology.orchestra.run.smithi094.stdout: "zones": [], 2021-09-06T12:02:37.386 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_targets": [], 2021-09-06T12:02:37.386 INFO:teuthology.orchestra.run.smithi094.stdout: "default_placement": "", 2021-09-06T12:02:37.386 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T12:02:37.386 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_policy": { 2021-09-06T12:02:37.387 INFO:teuthology.orchestra.run.smithi094.stdout: "groups": [] 2021-09-06T12:02:37.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:37.387 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T12:02:37.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:37.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:37.389 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:02:37.447 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:37.447 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:37.493 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T12:02:37.494 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "e80e23a1-41a2-4fa6-b832-56b972e2efcc", 2021-09-06T12:02:37.494 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "remove", 2021-09-06T12:02:37.494 INFO:teuthology.orchestra.run.smithi094.stdout: "domain_root": "remove.rgw.meta:root", 2021-09-06T12:02:37.494 INFO:teuthology.orchestra.run.smithi094.stdout: "control_pool": "remove.rgw.control", 2021-09-06T12:02:37.495 INFO:teuthology.orchestra.run.smithi094.stdout: "gc_pool": "remove.rgw.log:gc", 2021-09-06T12:02:37.495 INFO:teuthology.orchestra.run.smithi094.stdout: "lc_pool": "remove.rgw.log:lc", 2021-09-06T12:02:37.495 INFO:teuthology.orchestra.run.smithi094.stdout: "log_pool": "remove.rgw.log", 2021-09-06T12:02:37.495 INFO:teuthology.orchestra.run.smithi094.stdout: "intent_log_pool": "remove.rgw.log:intent", 2021-09-06T12:02:37.495 INFO:teuthology.orchestra.run.smithi094.stdout: "usage_log_pool": "remove.rgw.log:usage", 2021-09-06T12:02:37.496 INFO:teuthology.orchestra.run.smithi094.stdout: "roles_pool": "remove.rgw.meta:roles", 2021-09-06T12:02:37.496 INFO:teuthology.orchestra.run.smithi094.stdout: "reshard_pool": "remove.rgw.log:reshard", 2021-09-06T12:02:37.496 INFO:teuthology.orchestra.run.smithi094.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2021-09-06T12:02:37.496 INFO:teuthology.orchestra.run.smithi094.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2021-09-06T12:02:37.497 INFO:teuthology.orchestra.run.smithi094.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2021-09-06T12:02:37.497 INFO:teuthology.orchestra.run.smithi094.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2021-09-06T12:02:37.497 INFO:teuthology.orchestra.run.smithi094.stdout: "otp_pool": "remove.rgw.otp", 2021-09-06T12:02:37.497 INFO:teuthology.orchestra.run.smithi094.stdout: "system_key": { 2021-09-06T12:02:37.497 INFO:teuthology.orchestra.run.smithi094.stdout: "access_key": "", 2021-09-06T12:02:37.498 INFO:teuthology.orchestra.run.smithi094.stdout: "secret_key": "" 2021-09-06T12:02:37.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:37.498 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_pools": [ 2021-09-06T12:02:37.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:37.499 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "default-placement", 2021-09-06T12:02:37.499 INFO:teuthology.orchestra.run.smithi094.stdout: "val": { 2021-09-06T12:02:37.499 INFO:teuthology.orchestra.run.smithi094.stdout: "index_pool": "remove.rgw.buckets.index", 2021-09-06T12:02:37.499 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": { 2021-09-06T12:02:37.500 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD": { 2021-09-06T12:02:37.500 INFO:teuthology.orchestra.run.smithi094.stdout: "data_pool": "remove.rgw.buckets.data" 2021-09-06T12:02:37.500 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:37.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:37.501 INFO:teuthology.orchestra.run.smithi094.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2021-09-06T12:02:37.501 INFO:teuthology.orchestra.run.smithi094.stdout: "index_type": 0 2021-09-06T12:02:37.501 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:37.501 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:37.501 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:37.502 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T12:02:37.502 INFO:teuthology.orchestra.run.smithi094.stdout: "notif_pool": "remove.rgw.log:notif" 2021-09-06T12:02:37.502 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T12:02:37.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:37.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:37.504 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:02:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:37.552 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:38.019 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T12:02:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 6, 2021-09-06T12:02:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "predecessor_uuid": "8245bc5e-afe7-418b-afcf-7fd599759612", 2021-09-06T12:02:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": [], 2021-09-06T12:02:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "period_map": { 2021-09-06T12:02:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "zonegroups": [ 2021-09-06T12:02:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "208703f9-6eb4-4701-a7f2-dfe84295cfa0", 2021-09-06T12:02:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "remove_zg", 2021-09-06T12:02:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "api_name": "remove_zg", 2021-09-06T12:02:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "is_master": "false", 2021-09-06T12:02:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [], 2021-09-06T12:02:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames": [], 2021-09-06T12:02:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames_s3website": [], 2021-09-06T12:02:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "e80e23a1-41a2-4fa6-b832-56b972e2efcc", 2021-09-06T12:02:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "zones": [ 2021-09-06T12:02:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "e80e23a1-41a2-4fa6-b832-56b972e2efcc", 2021-09-06T12:02:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "remove", 2021-09-06T12:02:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [], 2021-09-06T12:02:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T12:02:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "false", 2021-09-06T12:02:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:02:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T12:02:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T12:02:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "true", 2021-09-06T12:02:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T12:02:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T12:02:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_targets": [ 2021-09-06T12:02:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default-placement", 2021-09-06T12:02:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2021-09-06T12:02:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": [ 2021-09-06T12:02:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD" 2021-09-06T12:02:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T12:02:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "default_placement": "default-placement", 2021-09-06T12:02:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T12:02:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_policy": { 2021-09-06T12:02:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "groups": [] 2021-09-06T12:02:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "test-zonegroup", 2021-09-06T12:02:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zonegroup", 2021-09-06T12:02:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "api_name": "", 2021-09-06T12:02:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "is_master": "true", 2021-09-06T12:02:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T12:02:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T12:02:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames": [], 2021-09-06T12:02:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames_s3website": [], 2021-09-06T12:02:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T12:02:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "zones": [ 2021-09-06T12:02:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T12:02:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone2", 2021-09-06T12:02:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T12:02:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2021-09-06T12:02:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T12:02:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T12:02:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:02:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T12:02:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T12:02:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "false", 2021-09-06T12:02:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T12:02:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T12:02:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "4487d356-c18b-4713-bd68-02dfa183d02a", 2021-09-06T12:02:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone4", 2021-09-06T12:02:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T12:02:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2021-09-06T12:02:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T12:02:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T12:02:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:02:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T12:02:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "pubsub", 2021-09-06T12:02:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "false", 2021-09-06T12:02:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [ 2021-09-06T12:02:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "test-zone1" 2021-09-06T12:02:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T12:02:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T12:02:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone1", 2021-09-06T12:02:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T12:02:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T12:02:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T12:02:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T12:02:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:02:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T12:02:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T12:02:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "true", 2021-09-06T12:02:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T12:02:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T12:02:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T12:02:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone3", 2021-09-06T12:02:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T12:02:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8001" 2021-09-06T12:02:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T12:02:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T12:02:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:02:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T12:02:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T12:02:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "true", 2021-09-06T12:02:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T12:02:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T12:02:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_targets": [ 2021-09-06T12:02:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default-placement", 2021-09-06T12:02:38.048 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2021-09-06T12:02:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": [ 2021-09-06T12:02:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD" 2021-09-06T12:02:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T12:02:38.049 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "default_placement": "default-placement", 2021-09-06T12:02:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T12:02:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_policy": { 2021-09-06T12:02:38.050 INFO:teuthology.orchestra.run.smithi094.stdout: "groups": [] 2021-09-06T12:02:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: "short_zone_ids": [ 2021-09-06T12:02:38.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T12:02:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 4134315483 2021-09-06T12:02:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.052 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "4487d356-c18b-4713-bd68-02dfa183d02a", 2021-09-06T12:02:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 2859952350 2021-09-06T12:02:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T12:02:38.053 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 772393743 2021-09-06T12:02:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T12:02:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 242578107 2021-09-06T12:02:38.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "e80e23a1-41a2-4fa6-b832-56b972e2efcc", 2021-09-06T12:02:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 3948081438 2021-09-06T12:02:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.055 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T12:02:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T12:02:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T12:02:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "period_config": { 2021-09-06T12:02:38.056 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_quota": { 2021-09-06T12:02:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T12:02:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T12:02:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T12:02:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T12:02:38.057 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T12:02:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "user_quota": { 2021-09-06T12:02:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T12:02:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T12:02:38.058 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T12:02:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T12:02:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T12:02:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.059 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T12:02:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_name": "test-realm", 2021-09-06T12:02:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T12:02:38.060 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T12:02:38.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:38.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:38.062 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:02:38.101 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:38.101 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:38.138 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:02:38.137+0000 7f44fc81f380 0 zone id e80e23a1-41a2-4fa6-b832-56b972e2efcc is not a part of zonegroup test-zonegroup 2021-09-06T12:02:38.139 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:02:38.138+0000 7f44fc81f380 0 zone id e80e23a1-41a2-4fa6-b832-56b972e2efcc is not a part of zonegroup default 2021-09-06T12:02:38.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:38.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:38.146 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:02:38.235 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:38.235 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:38.291 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:02:38.290+0000 7f567e763380 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-09-06T12:02:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-09-06T12:02:38.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:38.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:38.301 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:02:38.356 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:38.357 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:38.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:38.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:02:38.456 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:02:38.525 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:02:38.525 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:02:38.701 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2021-09-06T12:02:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: "epoch": 7, 2021-09-06T12:02:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: "predecessor_uuid": "8245bc5e-afe7-418b-afcf-7fd599759612", 2021-09-06T12:02:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_status": [], 2021-09-06T12:02:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: "period_map": { 2021-09-06T12:02:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "06970c73-91a9-4240-80e4-aa07d550699b", 2021-09-06T12:02:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: "zonegroups": [ 2021-09-06T12:02:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "test-zonegroup", 2021-09-06T12:02:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zonegroup", 2021-09-06T12:02:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: "api_name": "", 2021-09-06T12:02:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: "is_master": "true", 2021-09-06T12:02:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T12:02:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T12:02:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames": [], 2021-09-06T12:02:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: "hostnames_s3website": [], 2021-09-06T12:02:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T12:02:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: "zones": [ 2021-09-06T12:02:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T12:02:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone2", 2021-09-06T12:02:38.707 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T12:02:38.707 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2021-09-06T12:02:38.707 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.707 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T12:02:38.707 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T12:02:38.708 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:02:38.708 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T12:02:38.708 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T12:02:38.708 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "false", 2021-09-06T12:02:38.708 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T12:02:38.709 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T12:02:38.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.709 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "4487d356-c18b-4713-bd68-02dfa183d02a", 2021-09-06T12:02:38.710 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone4", 2021-09-06T12:02:38.710 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T12:02:38.710 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2021-09-06T12:02:38.710 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.710 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T12:02:38.711 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T12:02:38.711 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:02:38.711 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T12:02:38.711 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "pubsub", 2021-09-06T12:02:38.711 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "false", 2021-09-06T12:02:38.712 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [ 2021-09-06T12:02:38.712 INFO:teuthology.orchestra.run.smithi094.stdout: "test-zone1" 2021-09-06T12:02:38.712 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.712 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T12:02:38.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.713 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T12:02:38.713 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone1", 2021-09-06T12:02:38.713 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T12:02:38.714 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8000" 2021-09-06T12:02:38.714 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.714 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T12:02:38.714 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T12:02:38.714 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:02:38.715 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T12:02:38.715 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T12:02:38.715 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "true", 2021-09-06T12:02:38.715 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T12:02:38.715 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T12:02:38.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.716 INFO:teuthology.orchestra.run.smithi094.stdout: "id": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T12:02:38.716 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "test-zone3", 2021-09-06T12:02:38.717 INFO:teuthology.orchestra.run.smithi094.stdout: "endpoints": [ 2021-09-06T12:02:38.717 INFO:teuthology.orchestra.run.smithi094.stdout: "http://smithi094.front.sepia.ceph.com:8001" 2021-09-06T12:02:38.717 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.717 INFO:teuthology.orchestra.run.smithi094.stdout: "log_meta": "false", 2021-09-06T12:02:38.717 INFO:teuthology.orchestra.run.smithi094.stdout: "log_data": "true", 2021-09-06T12:02:38.718 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:02:38.718 INFO:teuthology.orchestra.run.smithi094.stdout: "read_only": "false", 2021-09-06T12:02:38.718 INFO:teuthology.orchestra.run.smithi094.stdout: "tier_type": "", 2021-09-06T12:02:38.718 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from_all": "true", 2021-09-06T12:02:38.719 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_from": [], 2021-09-06T12:02:38.719 INFO:teuthology.orchestra.run.smithi094.stdout: "redirect_zone": "" 2021-09-06T12:02:38.719 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.719 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.720 INFO:teuthology.orchestra.run.smithi094.stdout: "placement_targets": [ 2021-09-06T12:02:38.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.720 INFO:teuthology.orchestra.run.smithi094.stdout: "name": "default-placement", 2021-09-06T12:02:38.720 INFO:teuthology.orchestra.run.smithi094.stdout: "tags": [], 2021-09-06T12:02:38.721 INFO:teuthology.orchestra.run.smithi094.stdout: "storage_classes": [ 2021-09-06T12:02:38.721 INFO:teuthology.orchestra.run.smithi094.stdout: "STANDARD" 2021-09-06T12:02:38.721 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T12:02:38.721 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.722 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.722 INFO:teuthology.orchestra.run.smithi094.stdout: "default_placement": "default-placement", 2021-09-06T12:02:38.722 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T12:02:38.722 INFO:teuthology.orchestra.run.smithi094.stdout: "sync_policy": { 2021-09-06T12:02:38.722 INFO:teuthology.orchestra.run.smithi094.stdout: "groups": [] 2021-09-06T12:02:38.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.723 INFO:teuthology.orchestra.run.smithi094.stdout: ], 2021-09-06T12:02:38.723 INFO:teuthology.orchestra.run.smithi094.stdout: "short_zone_ids": [ 2021-09-06T12:02:38.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.733 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "2f90b352-42d8-4c14-ad62-fe843301400a", 2021-09-06T12:02:38.733 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 4134315483 2021-09-06T12:02:38.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.734 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "4487d356-c18b-4713-bd68-02dfa183d02a", 2021-09-06T12:02:38.734 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 2859952350 2021-09-06T12:02:38.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.735 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T12:02:38.735 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 772393743 2021-09-06T12:02:38.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2021-09-06T12:02:38.736 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "c248f771-5d9b-48a2-a793-2bc85cc1f251", 2021-09-06T12:02:38.736 INFO:teuthology.orchestra.run.smithi094.stdout: "val": 242578107 2021-09-06T12:02:38.736 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.736 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2021-09-06T12:02:38.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.737 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T12:02:38.737 INFO:teuthology.orchestra.run.smithi094.stdout: "master_zone": "7556a536-76a8-4c42-adbe-379e64c0e1a8", 2021-09-06T12:02:38.737 INFO:teuthology.orchestra.run.smithi094.stdout: "period_config": { 2021-09-06T12:02:38.737 INFO:teuthology.orchestra.run.smithi094.stdout: "bucket_quota": { 2021-09-06T12:02:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T12:02:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T12:02:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T12:02:38.738 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T12:02:38.739 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T12:02:38.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.739 INFO:teuthology.orchestra.run.smithi094.stdout: "user_quota": { 2021-09-06T12:02:38.739 INFO:teuthology.orchestra.run.smithi094.stdout: "enabled": false, 2021-09-06T12:02:38.739 INFO:teuthology.orchestra.run.smithi094.stdout: "check_on_raw": false, 2021-09-06T12:02:38.740 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size": -1, 2021-09-06T12:02:38.740 INFO:teuthology.orchestra.run.smithi094.stdout: "max_size_kb": 0, 2021-09-06T12:02:38.740 INFO:teuthology.orchestra.run.smithi094.stdout: "max_objects": -1 2021-09-06T12:02:38.740 INFO:teuthology.orchestra.run.smithi094.stdout: } 2021-09-06T12:02:38.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2021-09-06T12:02:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_id": "33774ddf-9586-4cf3-8e79-39a5aa159120", 2021-09-06T12:02:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_name": "test-realm", 2021-09-06T12:02:38.741 INFO:teuthology.orchestra.run.smithi094.stdout: "realm_epoch": 2 2021-09-06T12:02:38.741 INFO:teuthology.orchestra.run.smithi094.stdout:} 2021-09-06T12:02:38.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-09-06T12:02:38.743 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-46 2021-09-06T12:02:39.071 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ukmzsk-47 2021-09-06T12:02:39.296 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ukmzsk-48 2021-09-06T12:02:39.351 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-49 2021-09-06T12:02:39.353 INFO:tasks.rgw.c1.client.0.smithi094.stdout:*** Caught signal (Segmentation fault) ** 2021-09-06T12:02:39.354 INFO:tasks.rgw.c1.client.0.smithi094.stdout: in thread 7f98dd2fe700 thread_name:radosgw 2021-09-06T12:02:39.354 INFO:tasks.rgw.c1.client.0.smithi094.stdout: ceph version 17.0.0-7472-g2227f238 (2227f2382f1df5117a62687acd2d99ea5eeb871e) quincy (dev) 2021-09-06T12:02:39.354 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9965e63b20] 2021-09-06T12:02:39.354 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f9968a6067e] 2021-09-06T12:02:39.355 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7f9968a7e3e4] 2021-09-06T12:02:39.355 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 4: /lib64/libradosgw.so.2(+0x533b4d) [0x7f99689b1b4d] 2021-09-06T12:02:39.355 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 5: /lib64/libradosgw.so.2(+0x535130) [0x7f99689b3130] 2021-09-06T12:02:39.355 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 6: /lib64/libradosgw.so.2(+0x53528c) [0x7f99689b328c] 2021-09-06T12:02:39.356 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 7: make_fcontext() 2021-09-06T12:02:39.356 INFO:tasks.rgw.c1.client.0.smithi094.stdout:2021-09-06T12:02:39.352+0000 7f98dd2fe700 -1 *** Caught signal (Segmentation fault) ** 2021-09-06T12:02:39.356 INFO:tasks.rgw.c1.client.0.smithi094.stdout: in thread 7f98dd2fe700 thread_name:radosgw 2021-09-06T12:02:39.356 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 2021-09-06T12:02:39.357 INFO:tasks.rgw.c1.client.0.smithi094.stdout: ceph version 17.0.0-7472-g2227f238 (2227f2382f1df5117a62687acd2d99ea5eeb871e) quincy (dev) 2021-09-06T12:02:39.357 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9965e63b20] 2021-09-06T12:02:39.357 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f9968a6067e] 2021-09-06T12:02:39.357 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7f9968a7e3e4] 2021-09-06T12:02:39.357 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 4: /lib64/libradosgw.so.2(+0x533b4d) [0x7f99689b1b4d] 2021-09-06T12:02:39.358 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 5: /lib64/libradosgw.so.2(+0x535130) [0x7f99689b3130] 2021-09-06T12:02:39.358 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 6: /lib64/libradosgw.so.2(+0x53528c) [0x7f99689b328c] 2021-09-06T12:02:39.358 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 7: make_fcontext() 2021-09-06T12:02:39.358 INFO:tasks.rgw.c1.client.0.smithi094.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-06T12:02:39.359 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 2021-09-06T12:02:39.371 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 0> 2021-09-06T12:02:39.352+0000 7f98dd2fe700 -1 *** Caught signal (Segmentation fault) ** 2021-09-06T12:02:39.372 INFO:tasks.rgw.c1.client.0.smithi094.stdout: in thread 7f98dd2fe700 thread_name:radosgw 2021-09-06T12:02:39.372 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 2021-09-06T12:02:39.372 INFO:tasks.rgw.c1.client.0.smithi094.stdout: ceph version 17.0.0-7472-g2227f238 (2227f2382f1df5117a62687acd2d99ea5eeb871e) quincy (dev) 2021-09-06T12:02:39.373 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9965e63b20] 2021-09-06T12:02:39.373 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f9968a6067e] 2021-09-06T12:02:39.373 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7f9968a7e3e4] 2021-09-06T12:02:39.373 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 4: /lib64/libradosgw.so.2(+0x533b4d) [0x7f99689b1b4d] 2021-09-06T12:02:39.373 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 5: /lib64/libradosgw.so.2(+0x535130) [0x7f99689b3130] 2021-09-06T12:02:39.374 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 6: /lib64/libradosgw.so.2(+0x53528c) [0x7f99689b328c] 2021-09-06T12:02:39.374 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 7: make_fcontext() 2021-09-06T12:02:39.374 INFO:tasks.rgw.c1.client.0.smithi094.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-06T12:02:39.374 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 2021-09-06T12:02:39.392 INFO:tasks.rgw.c1.client.0.smithi094.stdout: -6> 2021-09-06T12:02:39.352+0000 7f98dd2fe700 -1 *** Caught signal (Segmentation fault) ** 2021-09-06T12:02:39.392 INFO:tasks.rgw.c1.client.0.smithi094.stdout: in thread 7f98dd2fe700 thread_name:radosgw 2021-09-06T12:02:39.392 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 2021-09-06T12:02:39.393 INFO:tasks.rgw.c1.client.0.smithi094.stdout: ceph version 17.0.0-7472-g2227f238 (2227f2382f1df5117a62687acd2d99ea5eeb871e) quincy (dev) 2021-09-06T12:02:39.393 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9965e63b20] 2021-09-06T12:02:39.393 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f9968a6067e] 2021-09-06T12:02:39.393 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7f9968a7e3e4] 2021-09-06T12:02:39.394 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 4: /lib64/libradosgw.so.2(+0x533b4d) [0x7f99689b1b4d] 2021-09-06T12:02:39.394 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 5: /lib64/libradosgw.so.2(+0x535130) [0x7f99689b3130] 2021-09-06T12:02:39.394 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 6: /lib64/libradosgw.so.2(+0x53528c) [0x7f99689b328c] 2021-09-06T12:02:39.394 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 7: make_fcontext() 2021-09-06T12:02:39.395 INFO:tasks.rgw.c1.client.0.smithi094.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-06T12:02:39.395 INFO:tasks.rgw.c1.client.0.smithi094.stdout: 2021-09-06T12:02:39.555 INFO:tasks.rgw.c1.client.0.smithi094.stderr:daemon-helper: command crashed with signal 11 2021-09-06T12:02:42.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-06T12:02:48.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2021-09-06T12:02:53.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~11s 2021-09-06T12:02:59.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~17s 2021-09-06T12:03:04.839 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~22s 2021-09-06T12:03:10.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~28s 2021-09-06T12:03:16.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~34s 2021-09-06T12:03:21.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~39s 2021-09-06T12:03:22.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... rgw_multi.tests.test_set_bucket_policy ... ERROR 2021-09-06T12:03:22.455 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-50 2021-09-06T12:03:27.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2021-09-06T12:03:32.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2021-09-06T12:03:38.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~56s 2021-09-06T12:03:44.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~62s 2021-09-06T12:03:49.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~67s 2021-09-06T12:03:55.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~73s 2021-09-06T12:04:00.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~78s 2021-09-06T12:04:06.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~84s 2021-09-06T12:04:12.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2021-09-06T12:04:17.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2021-09-06T12:04:23.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~101s 2021-09-06T12:04:26.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ERROR 2021-09-06T12:04:26.096 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-51 2021-09-06T12:04:28.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~107s 2021-09-06T12:04:34.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~112s 2021-09-06T12:04:40.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~118s 2021-09-06T12:04:45.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~123s 2021-09-06T12:04:51.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~129s 2021-09-06T12:04:56.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~135s 2021-09-06T12:05:02.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2021-09-06T12:05:08.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~146s 2021-09-06T12:05:13.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~151s 2021-09-06T12:05:19.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~157s 2021-09-06T12:05:24.982 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~163s 2021-09-06T12:05:30.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~168s 2021-09-06T12:05:36.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~174s 2021-09-06T12:05:41.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~179s 2021-09-06T12:05:43.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ERROR 2021-09-06T12:05:43.996 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-52 2021-09-06T12:05:47.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2021-09-06T12:05:53.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~191s 2021-09-06T12:05:58.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~196s 2021-09-06T12:06:04.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~202s 2021-09-06T12:06:09.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~207s 2021-09-06T12:06:15.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~213s 2021-09-06T12:06:21.037 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~219s 2021-09-06T12:06:23.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ERROR 2021-09-06T12:06:23.508 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-53 2021-09-06T12:06:26.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~224s 2021-09-06T12:06:32.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2021-09-06T12:06:37.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~235s 2021-09-06T12:06:43.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~241s 2021-09-06T12:06:49.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~247s 2021-09-06T12:06:54.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~252s 2021-09-06T12:07:00.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~258s 2021-09-06T12:07:05.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~263s 2021-09-06T12:07:11.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~269s 2021-09-06T12:07:17.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ERROR 2021-09-06T12:07:17.048 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-54 2021-09-06T12:07:17.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2021-09-06T12:07:22.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~280s 2021-09-06T12:07:28.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~286s 2021-09-06T12:07:33.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~291s 2021-09-06T12:07:39.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~297s 2021-09-06T12:07:45.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~303s 2021-09-06T12:07:45.115 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2021-09-06T12:07:45.720 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2021-09-06T12:07:45.722 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2021-09-06T12:07:45.722 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2021-09-06T12:07:45.723 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2021-09-06T12:07:45.723 INFO:tasks.rgw.c1.client.1:Sent signal 15 2021-09-06T12:07:45.724 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2021-09-06T12:07:45.725 INFO:tasks.ceph.c1.osd.0.smithi094.stderr:2021-09-06T12:07:45.720+0000 7f941d5b2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 59452) UID: 0 2021-09-06T12:07:45.725 INFO:tasks.ceph.c1.osd.0.smithi094.stderr:2021-09-06T12:07:45.720+0000 7f941d5b2700 -1 osd.0 61 *** Got signal Terminated *** 2021-09-06T12:07:45.725 INFO:tasks.ceph.c1.osd.0.smithi094.stderr:2021-09-06T12:07:45.720+0000 7f941d5b2700 -1 osd.0 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-06T12:07:45.726 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-09-06T12:07:45.722+0000 7fa02baeb700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 59456) UID: 0 2021-09-06T12:07:45.726 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-09-06T12:07:45.722+0000 7fa02baeb700 -1 osd.1 61 *** Got signal Terminated *** 2021-09-06T12:07:45.726 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-09-06T12:07:45.722+0000 7fa02baeb700 -1 osd.1 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-06T12:07:45.727 INFO:tasks.ceph.c1.osd.2.smithi094.stderr:2021-09-06T12:07:45.722+0000 7f7835b15700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 59458) UID: 0 2021-09-06T12:07:45.727 INFO:tasks.ceph.c1.osd.2.smithi094.stderr:2021-09-06T12:07:45.722+0000 7f7835b15700 -1 osd.2 61 *** Got signal Terminated *** 2021-09-06T12:07:45.727 INFO:tasks.ceph.c1.osd.2.smithi094.stderr:2021-09-06T12:07:45.722+0000 7f7835b15700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-06T12:07:45.728 INFO:tasks.ceph.c1.mon.a.smithi094.stderr:2021-09-06T12:07:45.723+0000 7f764dea2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 59135) UID: 0 2021-09-06T12:07:45.728 INFO:tasks.ceph.c1.mon.a.smithi094.stderr:2021-09-06T12:07:45.723+0000 7f764dea2700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-09-06T12:07:45.729 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T12:07:45.723+0000 7fc4d0247700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 67337) UID: 0 2021-09-06T12:07:45.729 INFO:tasks.ceph.c1.mgr.x.smithi094.stderr:2021-09-06T12:07:45.724+0000 7fc2c6437700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 59138) UID: 0 2021-09-06T12:07:45.730 INFO:tasks.ceph.c1.mgr.x.smithi094.stderr:2021-09-06T12:07:45.724+0000 7fc2c6437700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2021-09-06T12:07:45.802 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-09-06T12:07:45.723+0000 7fc4e10e5440 -1 shutting down 2021-09-06T12:08:39.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2021-09-06T12:08:39.605 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ukmzsk-55 2021-09-06T12:09:47.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2021-09-06T12:09:47.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:09:47.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:09:47.026 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:09:47.075 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:09:47.075 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:14:47.077 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:14:47.076+0000 7f058fe24380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T12:19:47.077 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:19:47.075+0000 7f058fe24380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T12:24:47.078 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:24:47.076+0000 7f058fe24380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T12:29:47.078 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:29:47.077+0000 7f058fe24380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T12:34:47.078 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:34:47.078+0000 7f058fe24380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T12:39:47.079 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:39:47.078+0000 7f058fe24380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T12:44:47.083 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:44:47.079+0000 7f058fe24380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T12:49:47.081 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:49:47.081+0000 7f058fe24380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T12:54:47.082 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:54:47.081+0000 7f058fe24380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T12:59:47.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T12:59:47.085 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T12:59:47.081+0000 7f058fe24380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T12:59:47.086 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T12:59:47.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2021-09-06T12:59:47.089 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-09-06T12:59:47.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:47.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:47.092 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:47.142 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:47.142 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:47.145 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:04:47.144+0000 7f08b090e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:09:47.145 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:09:47.144+0000 7f08b090e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:14:47.145 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:14:47.145+0000 7f08b090e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:19:47.146 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:19:47.145+0000 7f08b090e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:24:47.147 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:24:47.145+0000 7f08b090e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:29:47.148 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:29:47.146+0000 7f08b090e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:34:47.148 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:34:47.147+0000 7f08b090e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:39:47.149 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:39:47.147+0000 7f08b090e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:44:47.149 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:44:47.147+0000 7f08b090e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:49:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T13:49:47.153 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:49:47.148+0000 7f08b090e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:49:47.153 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T13:49:47.155 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2021-09-06T13:49:47.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:49:47.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:49:47.157 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:49:47.206 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:49:47.206 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:54:47.209 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:54:47.207+0000 7fdaf32a0380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:59:47.209 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T13:59:47.207+0000 7fdaf32a0380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:04:47.210 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:04:47.207+0000 7fdaf32a0380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:09:47.211 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:09:47.208+0000 7fdaf32a0380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:14:47.212 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:14:47.208+0000 7fdaf32a0380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:19:47.212 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:19:47.209+0000 7fdaf32a0380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:24:47.212 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:24:47.209+0000 7fdaf32a0380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:29:47.213 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:29:47.210+0000 7fdaf32a0380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:34:47.213 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:34:47.211+0000 7fdaf32a0380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:39:47.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T14:39:47.218 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:39:47.211+0000 7fdaf32a0380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:39:47.218 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T14:39:47.220 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2021-09-06T14:39:47.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T14:39:47.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T14:39:47.223 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T14:39:47.272 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T14:39:47.272 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T14:44:47.275 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:44:47.271+0000 7f0ad4761380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:49:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:49:47.273+0000 7f0ad4761380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:54:47.276 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:54:47.274+0000 7f0ad4761380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:59:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T14:59:47.273+0000 7f0ad4761380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:04:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:04:47.274+0000 7f0ad4761380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:09:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:09:47.275+0000 7f0ad4761380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:14:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:14:47.276+0000 7f0ad4761380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:19:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:19:47.275+0000 7f0ad4761380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:24:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:24:47.276+0000 7f0ad4761380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:29:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T15:29:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:29:47.277+0000 7f0ad4761380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:29:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T15:29:47.284 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2021-09-06T15:29:47.285 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2021-09-06T15:29:47.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2021-09-06T15:29:47.286 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-09-06T15:29:47.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T15:29:47.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T15:29:47.289 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T15:29:47.340 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T15:29:47.340 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T15:34:47.343 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:34:47.340+0000 7fba6c649380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:39:47.343 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:39:47.340+0000 7fba6c649380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:44:47.344 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:44:47.340+0000 7fba6c649380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:49:47.344 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:49:47.341+0000 7fba6c649380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:54:47.344 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:54:47.341+0000 7fba6c649380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:59:47.345 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T15:59:47.342+0000 7fba6c649380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:04:47.346 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:04:47.342+0000 7fba6c649380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:09:47.346 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:09:47.343+0000 7fba6c649380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:14:47.346 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:14:47.343+0000 7fba6c649380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:19:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T16:19:47.350 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:19:47.343+0000 7fba6c649380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:19:47.350 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T16:19:47.352 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2021-09-06T16:19:47.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T16:19:47.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T16:19:47.354 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T16:19:47.402 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T16:19:47.402 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T16:24:47.404 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:24:47.402+0000 7f25790e7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:29:47.405 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:29:47.403+0000 7f25790e7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:34:47.405 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:34:47.403+0000 7f25790e7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:39:47.406 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:39:47.403+0000 7f25790e7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:44:47.407 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:44:47.404+0000 7f25790e7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:49:47.407 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:49:47.405+0000 7f25790e7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:54:47.408 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:54:47.405+0000 7f25790e7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:59:47.408 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T16:59:47.406+0000 7f25790e7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:04:47.409 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:04:47.407+0000 7f25790e7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:09:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T17:09:47.412 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:09:47.407+0000 7f25790e7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:09:47.413 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T17:09:47.414 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2021-09-06T17:09:47.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T17:09:47.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T17:09:47.417 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T17:09:47.467 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T17:09:47.467 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T17:14:47.470 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:14:47.467+0000 7f89026a7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:19:47.470 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:19:47.468+0000 7f89026a7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:24:47.471 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:24:47.469+0000 7f89026a7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:29:47.471 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:29:47.469+0000 7f89026a7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:34:47.472 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:34:47.469+0000 7f89026a7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:39:47.472 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:39:47.470+0000 7f89026a7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:44:47.472 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:44:47.470+0000 7f89026a7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:49:47.473 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:49:47.471+0000 7f89026a7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:54:47.473 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:54:47.471+0000 7f89026a7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:59:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T17:59:47.477 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T17:59:47.471+0000 7f89026a7380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:59:47.477 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T17:59:47.479 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2021-09-06T17:59:47.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T17:59:47.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T17:59:47.481 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T17:59:47.531 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T17:59:47.531 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T18:04:47.533 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:04:47.531+0000 7fb8b013d380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:09:47.534 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:09:47.532+0000 7fb8b013d380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:14:47.534 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:14:47.532+0000 7fb8b013d380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:19:47.534 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:19:47.533+0000 7fb8b013d380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:24:47.535 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:24:47.533+0000 7fb8b013d380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:29:47.536 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:29:47.534+0000 7fb8b013d380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:34:47.536 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:34:47.534+0000 7fb8b013d380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:39:47.538 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:39:47.535+0000 7fb8b013d380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:44:47.538 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:44:47.536+0000 7fb8b013d380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:49:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T18:49:47.553 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:49:47.537+0000 7fb8b013d380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:49:47.553 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T18:49:47.555 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2021-09-06T18:49:47.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T18:49:47.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T18:49:47.557 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T18:49:47.608 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T18:49:47.609 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T18:54:47.611 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:54:47.609+0000 7ff6aefc1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:59:47.612 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T18:59:47.610+0000 7ff6aefc1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:04:47.612 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:04:47.611+0000 7ff6aefc1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:09:47.612 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:09:47.611+0000 7ff6aefc1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:14:47.613 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:14:47.611+0000 7ff6aefc1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:19:47.614 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:19:47.612+0000 7ff6aefc1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:24:47.614 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:24:47.612+0000 7ff6aefc1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:29:47.614 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:29:47.613+0000 7ff6aefc1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:34:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:34:47.613+0000 7ff6aefc1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:39:47.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T19:39:47.618 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:39:47.614+0000 7ff6aefc1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:39:47.618 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T19:39:47.619 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2021-09-06T19:39:47.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T19:39:47.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T19:39:47.622 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T19:39:47.670 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T19:39:47.671 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T19:44:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:44:47.672+0000 7f219c25c380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:49:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:49:47.673+0000 7f219c25c380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:54:47.674 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:54:47.672+0000 7f219c25c380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:59:47.675 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T19:59:47.673+0000 7f219c25c380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:04:47.676 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:04:47.674+0000 7f219c25c380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:09:47.678 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:09:47.675+0000 7f219c25c380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:14:47.677 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:14:47.675+0000 7f219c25c380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:19:47.677 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:19:47.675+0000 7f219c25c380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:24:47.678 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:24:47.676+0000 7f219c25c380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:29:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T20:29:47.709 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:29:47.677+0000 7f219c25c380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:29:47.709 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T20:29:47.711 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2021-09-06T20:29:47.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T20:29:47.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T20:29:47.715 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T20:29:47.767 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T20:29:47.767 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T20:34:47.769 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:34:47.769+0000 7ff7a51cd380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:39:47.770 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:39:47.769+0000 7ff7a51cd380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:44:47.772 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:44:47.769+0000 7ff7a51cd380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:49:47.774 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:49:47.771+0000 7ff7a51cd380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:54:47.773 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:54:47.772+0000 7ff7a51cd380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:59:47.774 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T20:59:47.772+0000 7ff7a51cd380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:04:47.774 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:04:47.773+0000 7ff7a51cd380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:09:47.775 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:09:47.774+0000 7ff7a51cd380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:14:47.776 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:14:47.774+0000 7ff7a51cd380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:19:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T21:19:47.791 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:19:47.774+0000 7ff7a51cd380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:19:47.792 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T21:19:47.793 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... FAIL 2021-09-06T21:19:47.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T21:19:47.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T21:19:47.796 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T21:19:47.845 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T21:19:47.846 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T21:24:47.848 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:24:47.847+0000 7fd4960eb380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:29:47.849 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:29:47.847+0000 7fd4960eb380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:34:47.851 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:34:47.848+0000 7fd4960eb380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:39:47.850 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:39:47.848+0000 7fd4960eb380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:44:47.851 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:44:47.848+0000 7fd4960eb380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:49:47.851 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:49:47.849+0000 7fd4960eb380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:54:47.852 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:54:47.849+0000 7fd4960eb380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:59:47.854 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T21:59:47.850+0000 7fd4960eb380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:04:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T22:04:47.850+0000 7fd4960eb380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:09:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T22:09:47.857 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T22:09:47.851+0000 7fd4960eb380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:09:47.857 INFO:teuthology.orchestra.run.smithi094.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T22:09:47.858 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... FAIL 2021-09-06T22:09:47.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T22:09:47.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T22:09:47.861 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T22:09:47.910 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-09-06T22:09:47.910 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T22:14:47.914 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T22:14:47.910+0000 7fbdefeb5380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:19:47.913 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T22:19:47.910+0000 7fbdefeb5380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:24:47.914 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T22:24:47.911+0000 7fbdefeb5380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:29:47.914 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T22:29:47.911+0000 7fbdefeb5380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:34:47.913 INFO:teuthology.orchestra.run.smithi094.stderr:2021-09-06T22:34:47.911+0000 7fbdefeb5380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:36:43.074 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-09-06T22:36:43.075 DEBUG:teuthology.task.console_log:Killing console logger for smithi094 2021-09-06T22:36:43.076 DEBUG:teuthology.task.console_log:Killing console logger for smithi139 2021-09-06T22:36:43.077 DEBUG:teuthology.exit:Finished running handlers